1
|
Giunti G, Doherty CP. Cocreating an Automated mHealth Apps Systematic Review Process With Generative AI: Design Science Research Approach. JMIR MEDICAL EDUCATION 2024; 10:e48949. [PMID: 38345839 PMCID: PMC10897815 DOI: 10.2196/48949] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 11/28/2023] [Accepted: 01/28/2024] [Indexed: 02/15/2024]
Abstract
BACKGROUND The use of mobile devices for delivering health-related services (mobile health [mHealth]) has rapidly increased, leading to a demand for summarizing the state of the art and practice through systematic reviews. However, the systematic review process is a resource-intensive and time-consuming process. Generative artificial intelligence (AI) has emerged as a potential solution to automate tedious tasks. OBJECTIVE This study aimed to explore the feasibility of using generative AI tools to automate time-consuming and resource-intensive tasks in a systematic review process and assess the scope and limitations of using such tools. METHODS We used the design science research methodology. The solution proposed is to use cocreation with a generative AI, such as ChatGPT, to produce software code that automates the process of conducting systematic reviews. RESULTS A triggering prompt was generated, and assistance from the generative AI was used to guide the steps toward developing, executing, and debugging a Python script. Errors in code were solved through conversational exchange with ChatGPT, and a tentative script was created. The code pulled the mHealth solutions from the Google Play Store and searched their descriptions for keywords that hinted toward evidence base. The results were exported to a CSV file, which was compared to the initial outputs of other similar systematic review processes. CONCLUSIONS This study demonstrates the potential of using generative AI to automate the time-consuming process of conducting systematic reviews of mHealth apps. This approach could be particularly useful for researchers with limited coding skills. However, the study has limitations related to the design science research methodology, subjectivity bias, and the quality of the search results used to train the language model.
Collapse
Affiliation(s)
- Guido Giunti
- Academic Unit of Neurology, School of Medicine, Trinity College Dublin, Dublin, Ireland
- Research Unit of Health Sciences and Technology, Faculty of Medicine, University of Oulu, Oulu, Finland
- FutureNeuro SFI Research Centre, Royal College of Surgeons in Ireland, Dublin, Ireland
| | - Colin P Doherty
- Academic Unit of Neurology, School of Medicine, Trinity College Dublin, Dublin, Ireland
- FutureNeuro SFI Research Centre, Royal College of Surgeons in Ireland, Dublin, Ireland
- Department of Neurology, St James Hospital, Dublin, Ireland
| |
Collapse
|
2
|
Hammer B, Virgili E, Bilotta F. Evidence-based literature review: De-duplication a cornerstone for quality. World J Methodol 2023; 13:390-398. [PMID: 38229943 PMCID: PMC10789108 DOI: 10.5662/wjm.v13.i5.390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/14/2023] [Revised: 11/03/2023] [Accepted: 11/29/2023] [Indexed: 12/20/2023] Open
Abstract
Evidence-based literature reviews play a vital role in contemporary research, facilitating the synthesis of knowledge from multiple sources to inform decision-making and scientific advancements. Within this framework, de-duplication emerges as a part of the process for ensuring the integrity and reliability of evidence extraction. This opinion review delves into the evolution of de-duplication, highlights its importance in evidence synthesis, explores various de-duplication methods, discusses evolving technologies, and proposes best practices. By addressing ethical considerations this paper emphasizes the significance of de-duplication as a cornerstone for quality in evidence-based literature reviews.
Collapse
Affiliation(s)
- Barbara Hammer
- Librarian at Medical Library, University of Bergen, Bergen 5020, Norway
| | - Elettra Virgili
- Anesthesiology, Critical Care and Pain Medicine, University of Rome “La Sapienza”, Rome 00166, Italy
| | - Federico Bilotta
- Anesthesiology, Critical Care and Pain Medicine, University of Rome “La Sapienza”, Rome 00166, Italy
| |
Collapse
|
3
|
Spooner C, Vivat B, White N, Bruun A, Rohde G, Kwek PX, Stone P. What outcomes do studies use to measure the impact of prognostication on people with advanced cancer? Findings from a systematic review of quantitative and qualitative studies. Palliat Med 2023; 37:1345-1364. [PMID: 37586031 PMCID: PMC10548779 DOI: 10.1177/02692163231191148] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 08/18/2023]
Abstract
BACKGROUND Studies evaluating the impact of prognostication in advanced cancer patients vary in the outcomes they measure, and there is a lack of consensus about which outcomes are most important. AIM To identify outcomes previously reported in prognostic research with people with advanced cancer, as a first step towards constructing a core outcome set for prognostic impact studies. DESIGN A systematic review was conducted and analysed in two subsets: one qualitative and one quantitative. (PROSPERO ID: CRD42022320117; 29/03/2022). DATA SOURCES Six databases were searched from inception to September 2022. We extracted data describing (1) outcomes used to measure the impact of prognostication and (2) patients' and informal caregivers' experiences and perceptions of prognostication in advanced cancer. We classified findings using the Core Outcome Measures in Effectiveness Trials (COMET) initiative taxonomy, along with a narrative description. We appraised retrieved studies for quality, but quality was not a basis for exclusion. RESULTS We identified 42 eligible studies: 32 quantitative, 6 qualitative, 4 mixed methods. We extracted 70 outcomes of prognostication in advanced cancer and organised them into 12 domains: (1) survival; (2) psychiatric outcomes; (3) general outcomes; (4) spiritual/religious/existential functioning/wellbeing, (5) emotional functioning/wellbeing; (6) cognitive functioning; (7) social functioning; (8) global quality of life; (9) delivery of care; (10) perceived health status; (11) personal circumstances; and (12) hospital/hospice use. CONCLUSION Outcome reporting and measurement varied markedly across the studies. A standardised approach to outcome reporting in studies of prognosis is necessary to enhance data synthesis, improve clinical practice and better align with stakeholders' priorities.
Collapse
Affiliation(s)
- Caitlin Spooner
- Marie Curie Palliative Care Research Department, University College London, London, UK
| | - Bella Vivat
- Marie Curie Palliative Care Research Department, University College London, London, UK
| | - Nicola White
- Marie Curie Palliative Care Research Department, University College London, London, UK
| | - Andrea Bruun
- Marie Curie Palliative Care Research Department, University College London, London, UK
| | - Gudrun Rohde
- Marie Curie Palliative Care Research Department, University College London, London, UK
- Faculty of Health and Sport Sciences, University of Agder, Kristiansand, Norway
| | - Pei Xing Kwek
- University College Dublin School of Medicine, University College Dublin, Dublin, Ireland
| | - Patrick Stone
- Marie Curie Palliative Care Research Department, University College London, London, UK
| |
Collapse
|
4
|
Hair K, Bahor Z, Macleod M, Liao J, Sena ES. The Automated Systematic Search Deduplicator (ASySD): a rapid, open-source, interoperable tool to remove duplicate citations in biomedical systematic reviews. BMC Biol 2023; 21:189. [PMID: 37674179 PMCID: PMC10483700 DOI: 10.1186/s12915-023-01686-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Accepted: 08/21/2023] [Indexed: 09/08/2023] Open
Abstract
BACKGROUND Researchers performing high-quality systematic reviews search across multiple databases to identify relevant evidence. However, the same publication is often retrieved from several databases. Identifying and removing such duplicates ("deduplication") can be extremely time-consuming, but failure to remove these citations can lead to the wrongful inclusion of duplicate data. Many existing tools are not sensitive enough, lack interoperability with other tools, are not freely accessible, or are difficult to use without programming knowledge. Here, we report the performance of our Automated Systematic Search Deduplicator (ASySD), a novel tool to perform automated deduplication of systematic searches for biomedical reviews. METHODS We evaluated ASySD's performance on 5 unseen biomedical systematic search datasets of various sizes (1845-79,880 citations). We compared the performance of ASySD with EndNote's automated deduplication option and with the Systematic Review Assistant Deduplication Module (SRA-DM). RESULTS ASySD identified more duplicates than either SRA-DM or EndNote, with a sensitivity in different datasets of 0.95 to 0.99. The false-positive rate was comparable to human performance, with a specificity of > 0.99. The tool took less than 1 h to identify and remove duplicates within each dataset. CONCLUSIONS For duplicate removal in biomedical systematic reviews, ASySD is a highly sensitive, reliable, and time-saving tool. It is open source and freely available online as both an R package and a user-friendly web application.
Collapse
Affiliation(s)
- Kaitlyn Hair
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK.
| | - Zsanett Bahor
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK
| | - Malcolm Macleod
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK
| | - Jing Liao
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK
| | - Emily S Sena
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK
| |
Collapse
|
5
|
Borissov N, Haas Q, Minder B, Kopp-Heim D, von Gernler M, Janka H, Teodoro D, Amini P. Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research. Syst Rev 2022; 11:172. [PMID: 35978441 PMCID: PMC9382798 DOI: 10.1186/s13643-022-02045-9] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/17/2022] [Accepted: 08/01/2022] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Identifying and removing reference duplicates when conducting systematic reviews (SRs) remain a major, time-consuming issue for authors who manually check for duplicates using built-in features in citation managers. To address issues related to manual deduplication, we developed an automated, efficient, and rapid artificial intelligence-based algorithm named Deduklick. Deduklick combines natural language processing algorithms with a set of rules created by expert information specialists. METHODS Deduklick's deduplication uses a multistep algorithm of data normalization, calculates a similarity score, and identifies unique and duplicate references based on metadata fields, such as title, authors, journal, DOI, year, issue, volume, and page number range. We measured and compared Deduklick's capacity to accurately detect duplicates with the information specialists' standard, manual duplicate removal process using EndNote on eight existing heterogeneous datasets. Using a sensitivity analysis, we manually cross-compared the efficiency and noise of both methods. DISCUSSION Deduklick achieved average recall of 99.51%, average precision of 100.00%, and average F1 score of 99.75%. In contrast, the manual deduplication process achieved average recall of 88.65%, average precision of 99.95%, and average F1 score of 91.98%. Deduklick achieved equal to higher expert-level performance on duplicate removal. It also preserved high metadata quality and drastically reduced time spent on analysis. Deduklick represents an efficient, transparent, ergonomic, and time-saving solution for identifying and removing duplicates in SRs searches. Deduklick could therefore simplify SRs production and represent important advantages for scientists, including saving time, increasing accuracy, reducing costs, and contributing to quality SRs.
Collapse
Affiliation(s)
- Nikolay Borissov
- Risklick AG, Spin-Off, University of Bern, Bern, Switzerland.,CTU Bern, University of Bern, Bern, Switzerland
| | - Quentin Haas
- Risklick AG, Spin-Off, University of Bern, Bern, Switzerland.,CTU Bern, University of Bern, Bern, Switzerland
| | - Beatrice Minder
- Public Health & Primary Care Library, University Library of Bern, University of Bern, Bern, Switzerland
| | - Doris Kopp-Heim
- Public Health & Primary Care Library, University Library of Bern, University of Bern, Bern, Switzerland
| | - Marc von Gernler
- Medical Library, University Library of Bern, University of Bern, Bern, Switzerland
| | - Heidrun Janka
- Medical Library, University Library of Bern, University of Bern, Bern, Switzerland
| | - Douglas Teodoro
- University of Applied Sciences and Arts Western Switzerland, Geneva, Switzerland.,Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland
| | - Poorya Amini
- Risklick AG, Spin-Off, University of Bern, Bern, Switzerland, . .,CTU Bern, University of Bern, Bern, Switzerland.
| |
Collapse
|
6
|
Aum S, Choe S. srBERT: automatic article classification model for systematic review using BERT. Syst Rev 2021; 10:285. [PMID: 34717768 PMCID: PMC8556883 DOI: 10.1186/s13643-021-01763-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/03/2020] [Accepted: 07/19/2021] [Indexed: 11/21/2022] Open
Abstract
BACKGROUND Systematic reviews (SRs) are recognized as reliable evidence, which enables evidence-based medicine to be applied to clinical practice. However, owing to the significant efforts required for an SR, its creation is time-consuming, which often leads to out-of-date results. To support SR tasks, tools for automating these SR tasks have been considered; however, applying a general natural language processing model to domain-specific articles and insufficient text data for training poses challenges. METHODS The research objective is to automate the classification of included articles using the Bidirectional Encoder Representations from Transformers (BERT) algorithm. In particular, srBERT models based on the BERT algorithm are pre-trained using abstracts of articles from two types of datasets, and the resulting model is then fine-tuned using the article titles. The performances of our proposed models are compared with those of existing general machine-learning models. RESULTS Our results indicate that the proposed srBERTmy model, pre-trained with abstracts of articles and a generated vocabulary, achieved state-of-the-art performance in both classification and relation-extraction tasks; for the first task, it achieved an accuracy of 94.35% (89.38%), F1 score of 66.12 (78.64), and area under the receiver operating characteristic curve of 0.77 (0.9) on the original and (generated) datasets, respectively. In the second task, the model achieved an accuracy of 93.5% with a loss of 27%, thereby outperforming the other evaluated models, including the original BERT model. CONCLUSIONS Our research shows the possibility of automatic article classification using machine-learning approaches to support SR tasks and its broad applicability. However, because the performance of our model depends on the size and class ratio of the training dataset, it is important to secure a dataset of sufficient quality, which may pose challenges.
Collapse
Affiliation(s)
- Sungmin Aum
- Institute of Science and Technology (KIST), 5, Hwarang-ro 14-gil, Seongbuk-gu, Seoul, Republic of Korea.
- Division of Nano and Information Technology, University of Science and Technology (UST), Gajeong-ro, Yuseong-gu, 34113, Daejeon, Republic of Korea.
- Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul, Republic of Korea.
- Data Republic, 320-1, Gwangnaru-ro, Seongdong-gu, Seoul, Republic of Korea.
| | - Seon Choe
- Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul National University Biomedical Informatics (SNUBI), Seoul, 03080, South Korea
| |
Collapse
|