Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews

Global Research on Pandemics or Epidemics and Mental Health: A Natural Language Processing Study

BACKGROUND

The global research on pandemics or epidemics and mental health has been growing exponentially recently, which cannot be integrated through traditional systematic review. Our study aims to systematically synthesize the evidence using natural language processing (NLP) techniques.

METHODS

Multiple databases were searched using titles, abstracts, and keywords. We systematically identified relevant literature published prior to Dec 31, 2023, using NLP techniques such as text classification, topic modelling and geoparsing methods. Relevant articles were categorized by content, date, and geographic location, outputting evidence heat maps, geographical maps, and narrative synthesis of trends in related publications.

RESULTS

Our NLP analysis identified 77,915 studies in the area of pandemics or epidemics and mental health published before Dec 31, 2023. The Covid pandemic was the most common, followed by SARS and HIV/AIDS; Anxiety and stress were the most frequently studied mental health outcomes; Social support and healthcare were the most common way of coping. Geographically, the evidence base was dominated by studies from high-income countries, with scant evidence from low-income counties. Co-occurrence of pandemics or epidemics and fear, depression, stress was common. Anxiety was one of the three most common topics in all continents except North America.

CONCLUSION

Our findings suggest the importance and feasibility of using NLP to comprehensively map pandemics or epidemics and mental health in the age of big literature. The review identifies clear themes for future clinical and public health research, and is critical for designing evidence-based approaches to reduce the negative mental health impacts of pandemics or epidemics.

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

BACKGROUND

The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice.

METHODS

In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies.

RESULTS

From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs.

CONCLUSIONS

Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Understanding the impact of sisu on workforce and well-being: A machine learning-based analysis

This study investigates the construct of sisu, a Finnish attribute representing mental resilience and fortitude when confronted with difficult situations. By leveraging advanced analytical methods and explainable Artificial Intelligence, we gain insights into how sisu factors influence well-being, work efficiency, and overall health. We investigate how the beneficial aspects of sisu contribute significantly to mental and physical health, satisfaction, and professional accomplishments. Conversely, we analyze the harmful sisu and its adverse impacts on the same domains. Our findings, including intriguing trends related to age, educational level, emotional states, and gender, pave the way for developing tailored solutions and initiatives to nurture the beneficial aspects of sisu and curtail the damaging consequences of sisu within professional settings and personal welfare.

What are the best methods for rapid reviews of the research evidence? A systematic review of reviews and primary studies

Rapid review methodology aims to facilitate faster conduct of systematic reviews to meet the needs of the decision-maker, while also maintaining quality and credibility. This systematic review aimed to determine the impact of different methodological shortcuts for undertaking rapid reviews on the risk of bias (RoB) of the results of the review. Review stages for which reviews and primary studies were sought included the preparation of a protocol, question formulation, inclusion criteria, searching, selection, data extraction, RoB assessment, synthesis, and reporting. We searched 11 electronic databases in April 2022, and conducted some supplementary searching. Reviewers worked in pairs to screen, select, extract data, and assess the RoB of included reviews and studies. We included 15 systematic reviews, 7 scoping reviews, and 65 primary studies. We found that several commonly used shortcuts in rapid reviews are likely to increase the RoB in the results. These include restrictions based on publication date, use of a single electronic database as a source of studies, and use of a single reviewer for screening titles and abstracts, selecting studies based on the full-text, and for extracting data. Authors of rapid reviews should be transparent in reporting their use of these shortcuts and acknowledge the possibility of them causing bias in the results. This review also highlights shortcuts that can save time without increasing the risk of bias. Further research is needed for both systematic and rapid reviews on faster methods for accurate data extraction and RoB assessment, and on development of more precise search strategies.

Semi-automating abstract screening with a natural language model pretrained on biomedical literature

We demonstrate the performance and workload impact of incorporating a natural language model, pretrained on citations of biomedical literature, on a workflow of abstract screening for studies on prognostic factors in end-stage lung disease. The model was optimized on one-third of the abstracts, and model performance on the remaining abstracts was reported. Performance of the model, in terms of sensitivity, precision, F1 and inter-rater agreement, was moderate in comparison with other published models. However, incorporating it into the screening workflow, with the second reviewer screening only abstracts with conflicting decisions, translated into a 65% reduction in the number of abstracts screened by the second reviewer. Subsequent work will look at incorporating the pre-trained BERT model into screening workflows for other studies prospectively, as well as improving model performance.

An open competition involving thousands of competitors failed to construct useful abstract classifiers for new diagnostic test accuracy systematic reviews

There are currently no abstract classifiers, which can be used for new diagnostic test accuracy (DTA) systematic reviews to select primary DTA study abstracts from database searches. Our goal was to develop machine-learning-based abstract classifiers for new DTA systematic reviews through an open competition. We prepared a dataset of abstracts obtained through database searches from 11 reviews in different clinical areas. As the reference standard, we used the abstract lists that required manual full-text review. We randomly splitted the datasets into a train set, a public test set, and a private test set. Competition participants used the training set to develop classifiers and validated their classifiers using the public test set. The classifiers were refined based on the performance of the public test set. They could submit as many times as they wanted during the competition. Finally, we used the private test set to rank the submitted classifiers. To reduce false exclusions, we used the Fbeta measure with a beta set to seven for evaluating classifiers. After the competition, we conducted the external validation using a dataset from a cardiology DTA review. We received 13,774 submissions from 1429 teams or persons over 4 months. The top-honored classifier achieved a Fbeta score of 0.4036 and a recall of 0.2352 in the external validation. In conclusion, we were unable to develop an abstract classifier with sufficient recall for immediate application to new DTA systematic reviews. Further studies are needed to update and validate classifiers with datasets from other clinical areas.

A real-world evaluation of the implementation of NLP technology in abstract screening of a systematic review

The laborious and time-consuming nature of systematic review production hinders the dissemination of up-to-date evidence synthesis. Well-performing natural language processing (NLP) tools for systematic reviews have been developed, showing promise to improve efficiency. However, the feasibility and value of these technologies have not been comprehensively demonstrated in a real-world review. We developed an NLP-assisted abstract screening tool that provides text inclusion recommendations, keyword highlights, and visual context cues. We evaluated this tool in a living systematic review on SARS-CoV-2 seroprevalence, conducting a quality improvement assessment of screening with and without the tool. We evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter-reviewer conflict rates. The tool conserved precision of article inclusion (positive predictive value; 0.92 with tool vs. 0.88 without) and recall (sensitivity; 0.90 vs. 0.81). The summary statistics of included studies were similar with and without the tool. Users were satisfied with the tool (mean satisfaction score of 4.2/5). We evaluated an abstract screening process where one human reviewer was replaced with the tool's votes, finding that this maintained recall (0.92 one-person, one-tool vs. 0.90 two tool-assisted humans) and precision (0.91 vs. 0.92) while reducing screening time by 70%. Implementing an NLP tool in this living systematic review improved efficiency, maintained accuracy, and was well-received by researchers, demonstrating the real-world effectiveness of NLP in expediting evidence synthesis.

The use of artificial intelligence for automating or semi-automating biomedical literature analyses: a scoping review

OBJECTIVE

Evidence-based medicine (EBM) is a decision-making process based on the conscious and judicious use of the best available scientific evidence. However, the exponential increase in the amount of information currently available likely exceeds the capacity of human-only analysis. In this context, artificial intelligence (AI) and its branches such as machine learning (ML) can be used to facilitate human efforts in analyzing the literature to foster EBM. The present scoping review aimed to examine the use of AI in the automation of biomedical literature survey and analysis with a view to establishing the state-of-the-art and identifying knowledge gaps.

MATERIALS AND METHODS

Comprehensive searches of the main databases were performed for articles published up to June 2022 and studies were selected according to inclusion and exclusion criteria. Data were extracted from the included articles and the findings categorized.

RESULTS

The total number of records retrieved from the databases was 12,145, of which 273 were included in the review. Classification of the studies according to the use of AI in evaluating the biomedical literature revealed three main application groups, namely assembly of scientific evidence (n=127; 47%), mining the biomedical literature (n=112; 41%) and quality analysis (n=34; 12%). Most studies addressed the preparation of systematic reviews, while articles focusing on the development of guidelines and evidence synthesis were the least frequent. The biggest knowledge gap was identified within the quality analysis group, particularly regarding methods and tools that assess the strength of recommendation and consistency of evidence.

CONCLUSION

Our review shows that, despite significant progress in the automation of biomedical literature surveys and analyses in recent years, intense research is needed to fill knowledge gaps on more difficult aspects of ML, deep learning and natural language processing, and to consolidate the use of automation by end-users (biomedical researchers and healthcare professionals).

Demystifying the Role of Natural Language Processing (NLP) in Smart City Applications: Background, Motivation, Recent Advances, and Future Research Directions

Smart cities provide an efficient infrastructure for the enhancement of the quality of life of the people by aiding in fast urbanization and resource management through sustainable and scalable innovative solutions. The penetration of Information and Communication Technology (ICT) in smart cities has been a major contributor to keeping up with the agility and pace of their development. In this paper, we have explored Natural Language Processing (NLP) which is one such technical discipline that has great potential in optimizing ICT processes and has so far been kept away from the limelight. Through this study, we have established the various roles that NLP plays in building smart cities after thoroughly analyzing its architecture, background, and scope. Subsequently, we present a detailed description of NLP's recent applications in the domain of smart healthcare, smart business, and industry, smart community, smart media, smart research, and development as well as smart education accompanied by NLP's open challenges at the very end. This work aims to throw light on the potential of NLP as one of the pillars in assisting the technical advancement and realization of smart cities.

NeuroCORD: A Language Model to Facilitate COVID-19-Associated Neurological Disorder Studies

COVID-19 can lead to multiple severe outcomes including neurological and psychological impacts. However, it is challenging to manually scan hundreds of thousands of COVID-19 articles on a regular basis. To update our knowledge, provide sound science to the public, and communicate effectively, it is critical to have an efficient means of following the most current published data. In this study, we developed a language model to search abstracts using the most advanced artificial intelligence (AI) to accurately retrieve articles on COVID-19-associated neurological disorders. We applied this NeuroCORD model to the largest benchmark dataset of COVID-19, CORD-19. We found that the model developed on the training set yielded 94% prediction accuracy on the test set. This result was subsequently verified by two experts in the field. In addition, when applied to 96,000 non-labeled articles that were published after 2020, the NeuroCORD model accurately identified approximately 3% of them to be relevant for the study of COVID-19-associated neurological disorders, while only 0.5% were retrieved using conventional keyword searching. In conclusion, NeuroCORD provides an opportunity to profile neurological disorders resulting from COVID-19 in a rapid and efficient fashion, and its general framework could be used to study other COVID-19-related emerging health issues.

Methodological guidance for rapid reviews in healthcare: a scoping review

OBJECTIVE

The aim of the present work was to identify published methodological guidance for rapid reviews (RRs) and to analyze the recommendations with regard to time-saving measures.

STUDY DESIGN AND SETTING

A literature search was performed in PubMed and EMBASE in November 2020. In addition, a search based on Google Scholar and websites of governmental and non-governmental organizations was conducted. Literature screening was carried out by two researchers independently.

RESULTS

A total of 34 publications were included. These describe 38 distinct RR types. The timeframe to complete the identified RR types ranges from 24 hours to six months (mean time 2.2 months). For most RR types a specific research question (n=21) and a prioritizing search (n=25; preference for e.g. systematic reviews, meta-analyses) is employed. Different approaches such as reduced personnel in literature screening (n=21) and data extraction (n=21) are recommended. The majority of RR types include a bias assessment (n=28) and suggest a narrative report focusing on safety and efficacy.

CONCLUSION

The included RR types are heterogeneous in terms of completion time, considered domains and strategies to alter the standard systematic review methods. A rationale for the recommended shortcuts is rarely presented. This article is protected by copyright. All rights reserved.

Tools to support the automation of systematic reviews: A scoping review

OBJECTIVE

The objectives of this scoping review are to identify the reliability and validity of the available tools, their limitations and any recommendations to further improve the use of these tools.

STUDY DESIGN

A scoping review methodology was followed to map the literature published on the challenges and solutions of conducting evidence synthesis using the JBI scoping review methodology.

RESULTS

A total of 47 publications were included in the review. The current scoping review identified that LitSuggest, Rayyan, Abstractr, BIBOT, R software, RobotAnalyst, DistillerSR, ExaCT and NetMetaXL have potential to be used for the automation of systematic reviews. However, they are not without limitations. The review also identified other studies that employed algorithms that have not yet been developed into user friendly tools. Some of these algorithms showed high validity and reliability but their use is conditional on user knowledge of computer science and algorithms.

CONCLUSION

Abstract screening has reached maturity; data extraction is still an active area. Developing methods to semi-automate different steps of evidence synthesis via machine learning remains an important research direction. Also, it is important to move from the research prototypes currently available to professionally maintained platforms.