Duplicates in systematic reviews: A critical, but often neglected issue

Cocreating an Automated mHealth Apps Systematic Review Process With Generative AI: Design Science Research Approach

BACKGROUND

The use of mobile devices for delivering health-related services (mobile health [mHealth]) has rapidly increased, leading to a demand for summarizing the state of the art and practice through systematic reviews. However, the systematic review process is a resource-intensive and time-consuming process. Generative artificial intelligence (AI) has emerged as a potential solution to automate tedious tasks.

OBJECTIVE

This study aimed to explore the feasibility of using generative AI tools to automate time-consuming and resource-intensive tasks in a systematic review process and assess the scope and limitations of using such tools.

METHODS

We used the design science research methodology. The solution proposed is to use cocreation with a generative AI, such as ChatGPT, to produce software code that automates the process of conducting systematic reviews.

RESULTS

A triggering prompt was generated, and assistance from the generative AI was used to guide the steps toward developing, executing, and debugging a Python script. Errors in code were solved through conversational exchange with ChatGPT, and a tentative script was created. The code pulled the mHealth solutions from the Google Play Store and searched their descriptions for keywords that hinted toward evidence base. The results were exported to a CSV file, which was compared to the initial outputs of other similar systematic review processes.

CONCLUSIONS

This study demonstrates the potential of using generative AI to automate the time-consuming process of conducting systematic reviews of mHealth apps. This approach could be particularly useful for researchers with limited coding skills. However, the study has limitations related to the design science research methodology, subjectivity bias, and the quality of the search results used to train the language model.

Evidence-based literature review: De-duplication a cornerstone for quality

Evidence-based literature reviews play a vital role in contemporary research, facilitating the synthesis of knowledge from multiple sources to inform decision-making and scientific advancements. Within this framework, de-duplication emerges as a part of the process for ensuring the integrity and reliability of evidence extraction. This opinion review delves into the evolution of de-duplication, highlights its importance in evidence synthesis, explores various de-duplication methods, discusses evolving technologies, and proposes best practices. By addressing ethical considerations this paper emphasizes the significance of de-duplication as a cornerstone for quality in evidence-based literature reviews.

What outcomes do studies use to measure the impact of prognostication on people with advanced cancer? Findings from a systematic review of quantitative and qualitative studies

BACKGROUND

Studies evaluating the impact of prognostication in advanced cancer patients vary in the outcomes they measure, and there is a lack of consensus about which outcomes are most important.

AIM

To identify outcomes previously reported in prognostic research with people with advanced cancer, as a first step towards constructing a core outcome set for prognostic impact studies.

DESIGN

A systematic review was conducted and analysed in two subsets: one qualitative and one quantitative. (PROSPERO ID: CRD42022320117; 29/03/2022).

DATA SOURCES

Six databases were searched from inception to September 2022. We extracted data describing (1) outcomes used to measure the impact of prognostication and (2) patients' and informal caregivers' experiences and perceptions of prognostication in advanced cancer. We classified findings using the Core Outcome Measures in Effectiveness Trials (COMET) initiative taxonomy, along with a narrative description. We appraised retrieved studies for quality, but quality was not a basis for exclusion.

RESULTS

We identified 42 eligible studies: 32 quantitative, 6 qualitative, 4 mixed methods. We extracted 70 outcomes of prognostication in advanced cancer and organised them into 12 domains: (1) survival; (2) psychiatric outcomes; (3) general outcomes; (4) spiritual/religious/existential functioning/wellbeing, (5) emotional functioning/wellbeing; (6) cognitive functioning; (7) social functioning; (8) global quality of life; (9) delivery of care; (10) perceived health status; (11) personal circumstances; and (12) hospital/hospice use.

CONCLUSION

Outcome reporting and measurement varied markedly across the studies. A standardised approach to outcome reporting in studies of prognosis is necessary to enhance data synthesis, improve clinical practice and better align with stakeholders' priorities.

The Automated Systematic Search Deduplicator (ASySD): a rapid, open-source, interoperable tool to remove duplicate citations in biomedical systematic reviews

BACKGROUND

Researchers performing high-quality systematic reviews search across multiple databases to identify relevant evidence. However, the same publication is often retrieved from several databases. Identifying and removing such duplicates ("deduplication") can be extremely time-consuming, but failure to remove these citations can lead to the wrongful inclusion of duplicate data. Many existing tools are not sensitive enough, lack interoperability with other tools, are not freely accessible, or are difficult to use without programming knowledge. Here, we report the performance of our Automated Systematic Search Deduplicator (ASySD), a novel tool to perform automated deduplication of systematic searches for biomedical reviews.

METHODS

We evaluated ASySD's performance on 5 unseen biomedical systematic search datasets of various sizes (1845-79,880 citations). We compared the performance of ASySD with EndNote's automated deduplication option and with the Systematic Review Assistant Deduplication Module (SRA-DM).

RESULTS

ASySD identified more duplicates than either SRA-DM or EndNote, with a sensitivity in different datasets of 0.95 to 0.99. The false-positive rate was comparable to human performance, with a specificity of > 0.99. The tool took less than 1 h to identify and remove duplicates within each dataset.

CONCLUSIONS

For duplicate removal in biomedical systematic reviews, ASySD is a highly sensitive, reliable, and time-saving tool. It is open source and freely available online as both an R package and a user-friendly web application.

Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research

BACKGROUND

Identifying and removing reference duplicates when conducting systematic reviews (SRs) remain a major, time-consuming issue for authors who manually check for duplicates using built-in features in citation managers. To address issues related to manual deduplication, we developed an automated, efficient, and rapid artificial intelligence-based algorithm named Deduklick. Deduklick combines natural language processing algorithms with a set of rules created by expert information specialists.

METHODS

Deduklick's deduplication uses a multistep algorithm of data normalization, calculates a similarity score, and identifies unique and duplicate references based on metadata fields, such as title, authors, journal, DOI, year, issue, volume, and page number range. We measured and compared Deduklick's capacity to accurately detect duplicates with the information specialists' standard, manual duplicate removal process using EndNote on eight existing heterogeneous datasets. Using a sensitivity analysis, we manually cross-compared the efficiency and noise of both methods.

DISCUSSION

Deduklick achieved average recall of 99.51%, average precision of 100.00%, and average F1 score of 99.75%. In contrast, the manual deduplication process achieved average recall of 88.65%, average precision of 99.95%, and average F1 score of 91.98%. Deduklick achieved equal to higher expert-level performance on duplicate removal. It also preserved high metadata quality and drastically reduced time spent on analysis. Deduklick represents an efficient, transparent, ergonomic, and time-saving solution for identifying and removing duplicates in SRs searches. Deduklick could therefore simplify SRs production and represent important advantages for scientists, including saving time, increasing accuracy, reducing costs, and contributing to quality SRs.

srBERT: automatic article classification model for systematic review using BERT

BACKGROUND

Systematic reviews (SRs) are recognized as reliable evidence, which enables evidence-based medicine to be applied to clinical practice. However, owing to the significant efforts required for an SR, its creation is time-consuming, which often leads to out-of-date results. To support SR tasks, tools for automating these SR tasks have been considered; however, applying a general natural language processing model to domain-specific articles and insufficient text data for training poses challenges.

METHODS

The research objective is to automate the classification of included articles using the Bidirectional Encoder Representations from Transformers (BERT) algorithm. In particular, srBERT models based on the BERT algorithm are pre-trained using abstracts of articles from two types of datasets, and the resulting model is then fine-tuned using the article titles. The performances of our proposed models are compared with those of existing general machine-learning models.

RESULTS

Our results indicate that the proposed srBERTmy model, pre-trained with abstracts of articles and a generated vocabulary, achieved state-of-the-art performance in both classification and relation-extraction tasks; for the first task, it achieved an accuracy of 94.35% (89.38%), F1 score of 66.12 (78.64), and area under the receiver operating characteristic curve of 0.77 (0.9) on the original and (generated) datasets, respectively. In the second task, the model achieved an accuracy of 93.5% with a loss of 27%, thereby outperforming the other evaluated models, including the original BERT model.

CONCLUSIONS

Our research shows the possibility of automatic article classification using machine-learning approaches to support SR tasks and its broad applicability. However, because the performance of our model depends on the size and class ratio of the training dataset, it is important to secure a dataset of sufficient quality, which may pose challenges.