1
|
Schlicht IB, Fernandez E, Chulvi B, Rosso P. Automatic detection of health misinformation: a systematic review. JOURNAL OF AMBIENT INTELLIGENCE AND HUMANIZED COMPUTING 2023:1-13. [PMID: 37360776 PMCID: PMC10220340 DOI: 10.1007/s12652-023-04619-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/30/2022] [Accepted: 04/30/2023] [Indexed: 06/28/2023]
Abstract
The spread of health misinformation has the potential to cause serious harm to public health, from leading to vaccine hesitancy to adoption of unproven disease treatments. In addition, it could have other effects on society such as an increase in hate speech towards ethnic groups or medical experts. To counteract the sheer amount of misinformation, there is a need to use automatic detection methods. In this paper we conduct a systematic review of the computer science literature exploring text mining techniques and machine learning methods to detect health misinformation. To organize the reviewed papers, we propose a taxonomy, examine publicly available datasets, and conduct a content-based analysis to investigate analogies and differences among Covid-19 datasets and datasets related to other health domains. Finally, we describe open challenges and conclude with future directions.
Collapse
Affiliation(s)
| | | | - Berta Chulvi
- Universitat Politècnica de València, Valencia, Spain
| | - Paolo Rosso
- Universitat Politècnica de València, Valencia, Spain
| |
Collapse
|
2
|
Liu X, Alsghaier H, Tong L, Ataullah A, McRoy S. Visualizing the Interpretation of a Criteria-Driven System That Automatically Evaluates the Quality of Health News: Exploratory Study of 2 Approaches. JMIR AI 2022; 1:e37751. [PMID: 38875559 PMCID: PMC11041450 DOI: 10.2196/37751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/04/2022] [Revised: 09/22/2022] [Accepted: 11/11/2022] [Indexed: 06/16/2024]
Abstract
BACKGROUND Machine learning techniques have been shown to be efficient in identifying health misinformation, but the results may not be trusted unless they can be justified in a way that is understandable. OBJECTIVE This study aimed to provide a new criteria-based system to assess and justify health news quality. Using a subset of an existing set of criteria, this study compared the feasibility of 2 alternative methods for adding interpretability. Both methods used classification and highlighting to visualize sentence-level evidence. METHODS A total of 3 out of 10 well-established criteria were chosen for experimentation, namely whether the health news discussed the costs of the intervention (the cost criterion), explained or quantified the harms of the intervention (the harm criterion), and identified the conflicts of interest (the conflict criterion). The first step of the experiment was to automate the evaluation of the 3 criteria by developing a sentence-level classifier. We tested Logistic Regression, Naive Bayes, Support Vector Machine, and Random Forest algorithms. Next, we compared the 2 visualization approaches. For the first approach, we calculated word feature weights, which explained how classification models distill keywords that contribute to the prediction; then, using the local interpretable model-agnostic explanation framework, we selected keywords associated with the classified criterion at the document level; and finally, the system selected and highlighted sentences with keywords. For the second approach, we extracted sentences that provided evidence to support the evaluation result from 100 health news articles; based on these results, we trained a typology classification model at the sentence level; and then, the system highlighted a positive sentence instance for the result justification. The number of sentences to highlight was determined by a preset threshold empirically determined using the average accuracy. RESULTS The automatic evaluation of health news on the cost, harm, and conflict criteria achieved average area under the curve scores of 0.88, 0.76, and 0.73, respectively, after 50 repetitions of 10-fold cross-validation. We found that both approaches could successfully visualize the interpretation of the system but that the performance of the 2 approaches varied by criterion and highlighting the accuracy decreased as the number of highlighted sentences increased. When the threshold accuracy was ≥75%, this resulted in a visualization with a variable length ranging from 1 to 6 sentences. CONCLUSIONS We provided 2 approaches to interpret criteria-based health news evaluation models tested on 3 criteria. This method incorporated rule-based and statistical machine learning approaches. The results suggested that one might visually interpret an automatic criterion-based health news quality evaluation successfully using either approach; however, larger differences may arise when multiple quality-related criteria are considered. This study can increase public trust in computerized health information evaluation.
Collapse
Affiliation(s)
- Xiaoyu Liu
- Department of Computer Science, University of Wisconsin Milwaukee, Milwaukee, WI, United States
- School of Health Sciences, Southern Illinois University Carbondale, Carbondale, IL, United States
| | - Hiba Alsghaier
- Department of Computer Science, University of Wisconsin Milwaukee, Milwaukee, WI, United States
| | - Ling Tong
- Department of Health Informatics and Administration, University of Wisconsin Milwaukee, Milwaukee, WI, United States
| | - Amna Ataullah
- Department of Computer Science, University of Wisconsin Milwaukee, Milwaukee, WI, United States
| | - Susan McRoy
- Department of Computer Science, University of Wisconsin Milwaukee, Milwaukee, WI, United States
| |
Collapse
|
3
|
Nabożny A, Balcerzak B, Morzy M, Wierzbicki A, Savov P, Warpechowski K. Improving medical experts' efficiency of misinformation detection: an exploratory study. WORLD WIDE WEB 2022; 26:773-798. [PMID: 35975112 PMCID: PMC9371952 DOI: 10.1007/s11280-022-01084-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/19/2022] [Revised: 05/03/2022] [Accepted: 07/04/2022] [Indexed: 06/15/2023]
Abstract
Fighting medical disinformation in the era of the pandemic is an increasingly important problem. Today, automatic systems for assessing the credibility of medical information do not offer sufficient precision, so human supervision and the involvement of medical expert annotators are required. Our work aims to optimize the utilization of medical experts' time. We also equip them with tools for semi-automatic initial verification of the credibility of the annotated content. We introduce a general framework for filtering medical statements that do not require manual evaluation by medical experts, thus focusing annotation efforts on non-credible medical statements. Our framework is based on the construction of filtering classifiers adapted to narrow thematic categories. This allows medical experts to fact-check and identify over two times more non-credible medical statements in a given time interval without applying any changes to the annotation flow. We verify our results across a broad spectrum of medical topic areas. We perform quantitative, as well as exploratory analysis on our output data. We also point out how those filtering classifiers can be modified to provide experts with different types of feedback without any loss of performance.
Collapse
Affiliation(s)
| | | | - Mikołaj Morzy
- Polish-Japanese Academy of Information Technology, Warsaw, Poland
- Poznań University of Technology, Poznań, Poland
| | - Adam Wierzbicki
- Polish-Japanese Academy of Information Technology, Warsaw, Poland
| | - Pavel Savov
- Polish-Japanese Academy of Information Technology, Warsaw, Poland
| | | |
Collapse
|
4
|
Kantor D, Farlow M, Ludolph A, Montaner J, Sankar R, Sawyer R, Stocchi F, Lara A, Clark S, Ouyahia L, Deschet K, Hadjiat Y. Digital Neurology Platform: Developing and implementing a rigorous content quality guideline. Interact J Med Res 2022; 11:e35698. [PMID: 35485280 PMCID: PMC9227648 DOI: 10.2196/35698] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2021] [Revised: 01/14/2022] [Accepted: 04/26/2022] [Indexed: 11/13/2022] Open
Abstract
Background Digital communication has emerged as a major source of scientific and medical information for health care professionals. There is a need to set up an effective and reliable methodology to assess and monitor the quality of content that is published on the internet. Objective The aim of this project was to develop content quality guidelines for Neurodiem, an independent scientific information platform dedicated to neurology for health care professionals and neuroscientists. These content quality guidelines are intended to be used by (1) content providers as a framework to meet content quality standards and (2) reviewers as a tool for analyzing and scoring quality of content. Methods Specific scientific criteria were designed using a 5-point scale to measure the quality of curated and original content published on the website: for Summaries, (1) source reliability and topic relevance for neurologists, (2) structure, and (3) scientific and didactic value; for Congress highlights, (1) relevance of congress selection, (2) congress coverage based on the original program, and (3) scientific and didactic value of individual abstracts; for Expert points of view and talks, (1) credibility (authorship) and topic relevance for neurologists, (2) scientific and didactic value, and (3) reliability (references) and format. The criteria were utilized on a monthly basis and endorsed by an independent scientific committee of widely recognized medical experts in neurology. Results Summary content quality for the 3 domains (reliability and relevance, structure, and scientific and didactic value) increased in the second month after the implementation of the guidelines. The domain scientific and didactic value had a mean score of 8.20/10. Scores for the domains reliability and relevance (8-9/10) and structure (45-55/60) showed that the maintenance of these 2 quality items over time was more challenging. Talks (either in the format of interviews or slide deck–supported scientific presentations) and expert point of view demonstrated high quality after the implementation of the content quality guidelines that was maintained over time (15-25/25). Conclusions Our findings support that content quality guidelines provide both (1) a reliable framework for generating independent high-quality content that addresses the educational needs of neurologists and (2) are an objective evaluation tool for improving and maintaining scientific quality level. The use of these criteria and this scoring system could serve as a standard and reference to build an editorial strategy and review process for any medical news or platforms.
Collapse
Affiliation(s)
- Daniel Kantor
- Florida Atlantic University, Boca Raton, FL, USA and Nova Southeastern University, Fort Lauderdale, FL, USA, Fort Lauderdale, US
| | - Martin Farlow
- Department of Neurology, Indiana University School of Medicine, Indianapolis, IN, USA, indianapolis, US
| | - Albert Ludolph
- Department of Neurology, University of Ulm, DZNE, Ulm, Germany, Ulm, DE
| | - Joan Montaner
- Department of Neurology, Hospital Universitario Virgen Macarena, Seville, Spain., Seville, ES
| | - Roman Sankar
- Division of Neurology, Department of Pediatrics, UCLA Mattel Children's Hospital, USA; Department of Neurology, David Geffen School of Medicine at UCLA, USA, ucla, US
| | - Robert Sawyer
- Department of Neurology, University at Buffalo, State University of New York, Buffalo, NY, USA, new york, US
| | - Fabrizio Stocchi
- University and Institute for Research and Medical Care, IRCCS San Raffaele, Rome, Italy., Rome, IT
| | - Agnès Lara
- Medicom concept, Llupia, Occitanie, France, Occitanie, FR
| | - Sarah Clark
- Biogen Digital Health, 225 Binney StreetBiogen, Cambridge, US
| | - Loucif Ouyahia
- Biogen Digital Health, 225 Binney StreetBiogen, Cambridge, US
| | - Karine Deschet
- Biogen Digital Health, 225 Binney StreetBiogen, Cambridge, US
| | | |
Collapse
|
5
|
Nabożny A, Balcerzak B, Wierzbicki A, Morzy M, Chlabicz M. Active Annotation in Evaluating the Credibility of Web-Based Medical Information: Guidelines for Creating Training Data Sets for Machine Learning. JMIR Med Inform 2021; 9:e26065. [PMID: 34842547 PMCID: PMC8665397 DOI: 10.2196/26065] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2020] [Revised: 03/29/2021] [Accepted: 09/24/2021] [Indexed: 01/14/2023] Open
Abstract
Background The spread of false medical information on the web is rapidly accelerating. Establishing the credibility of web-based medical information has become a pressing necessity. Machine learning offers a solution that, when properly deployed, can be an effective tool in fighting medical misinformation on the web. Objective The aim of this study is to present a comprehensive framework for designing and curating machine learning training data sets for web-based medical information credibility assessment. We show how to construct the annotation process. Our main objective is to support researchers from the medical and computer science communities. We offer guidelines on the preparation of data sets for machine learning models that can fight medical misinformation. Methods We begin by providing the annotation protocol for medical experts involved in medical sentence credibility evaluation. The protocol is based on a qualitative study of our experimental data. To address the problem of insufficient initial labels, we propose a preprocessing pipeline for the batch of sentences to be assessed. It consists of representation learning, clustering, and reranking. We call this process active annotation. Results We collected more than 10,000 annotations of statements related to selected medical subjects (psychiatry, cholesterol, autism, antibiotics, vaccines, steroids, birth methods, and food allergy testing) for less than US $7000 by employing 9 highly qualified annotators (certified medical professionals), and we release this data set to the general public. We developed an active annotation framework for more efficient annotation of noncredible medical statements. The application of qualitative analysis resulted in a better annotation protocol for our future efforts in data set creation. Conclusions The results of the qualitative analysis support our claims of the efficacy of the presented method.
Collapse
Affiliation(s)
- Aleksandra Nabożny
- Department of Software Engineering, Gdańsk University of Technology, Gdańsk, Poland
| | | | - Adam Wierzbicki
- Polish-Japanese Academy of Information Technology, Warsaw, Poland
| | - Mikołaj Morzy
- Faculty of Computing and Telecommunications, Poznan University of Technology, Poznań, Poland
| | - Małgorzata Chlabicz
- Department of Population Medicine and Lifestyle Diseases Prevention, Medical University of Białystok, Białystok, Poland
| |
Collapse
|