Automatically Assessing Quality of Online Health Articles

Automatic detection of health misinformation: a systematic review

The spread of health misinformation has the potential to cause serious harm to public health, from leading to vaccine hesitancy to adoption of unproven disease treatments. In addition, it could have other effects on society such as an increase in hate speech towards ethnic groups or medical experts. To counteract the sheer amount of misinformation, there is a need to use automatic detection methods. In this paper we conduct a systematic review of the computer science literature exploring text mining techniques and machine learning methods to detect health misinformation. To organize the reviewed papers, we propose a taxonomy, examine publicly available datasets, and conduct a content-based analysis to investigate analogies and differences among Covid-19 datasets and datasets related to other health domains. Finally, we describe open challenges and conclude with future directions.

Visualizing the Interpretation of a Criteria-Driven System That Automatically Evaluates the Quality of Health News: Exploratory Study of 2 Approaches

BACKGROUND

Machine learning techniques have been shown to be efficient in identifying health misinformation, but the results may not be trusted unless they can be justified in a way that is understandable.

OBJECTIVE

This study aimed to provide a new criteria-based system to assess and justify health news quality. Using a subset of an existing set of criteria, this study compared the feasibility of 2 alternative methods for adding interpretability. Both methods used classification and highlighting to visualize sentence-level evidence.

METHODS

A total of 3 out of 10 well-established criteria were chosen for experimentation, namely whether the health news discussed the costs of the intervention (the cost criterion), explained or quantified the harms of the intervention (the harm criterion), and identified the conflicts of interest (the conflict criterion). The first step of the experiment was to automate the evaluation of the 3 criteria by developing a sentence-level classifier. We tested Logistic Regression, Naive Bayes, Support Vector Machine, and Random Forest algorithms. Next, we compared the 2 visualization approaches. For the first approach, we calculated word feature weights, which explained how classification models distill keywords that contribute to the prediction; then, using the local interpretable model-agnostic explanation framework, we selected keywords associated with the classified criterion at the document level; and finally, the system selected and highlighted sentences with keywords. For the second approach, we extracted sentences that provided evidence to support the evaluation result from 100 health news articles; based on these results, we trained a typology classification model at the sentence level; and then, the system highlighted a positive sentence instance for the result justification. The number of sentences to highlight was determined by a preset threshold empirically determined using the average accuracy.

RESULTS

The automatic evaluation of health news on the cost, harm, and conflict criteria achieved average area under the curve scores of 0.88, 0.76, and 0.73, respectively, after 50 repetitions of 10-fold cross-validation. We found that both approaches could successfully visualize the interpretation of the system but that the performance of the 2 approaches varied by criterion and highlighting the accuracy decreased as the number of highlighted sentences increased. When the threshold accuracy was ≥75%, this resulted in a visualization with a variable length ranging from 1 to 6 sentences.

CONCLUSIONS

We provided 2 approaches to interpret criteria-based health news evaluation models tested on 3 criteria. This method incorporated rule-based and statistical machine learning approaches. The results suggested that one might visually interpret an automatic criterion-based health news quality evaluation successfully using either approach; however, larger differences may arise when multiple quality-related criteria are considered. This study can increase public trust in computerized health information evaluation.

Improving medical experts' efficiency of misinformation detection: an exploratory study

Fighting medical disinformation in the era of the pandemic is an increasingly important problem. Today, automatic systems for assessing the credibility of medical information do not offer sufficient precision, so human supervision and the involvement of medical expert annotators are required. Our work aims to optimize the utilization of medical experts' time. We also equip them with tools for semi-automatic initial verification of the credibility of the annotated content. We introduce a general framework for filtering medical statements that do not require manual evaluation by medical experts, thus focusing annotation efforts on non-credible medical statements. Our framework is based on the construction of filtering classifiers adapted to narrow thematic categories. This allows medical experts to fact-check and identify over two times more non-credible medical statements in a given time interval without applying any changes to the annotation flow. We verify our results across a broad spectrum of medical topic areas. We perform quantitative, as well as exploratory analysis on our output data. We also point out how those filtering classifiers can be modified to provide experts with different types of feedback without any loss of performance.

Digital Neurology Platform: Developing and implementing a rigorous content quality guideline

Background

Digital communication has emerged as a major source of scientific and medical information for health care professionals. There is a need to set up an effective and reliable methodology to assess and monitor the quality of content that is published on the internet.

Objective

The aim of this project was to develop content quality guidelines for Neurodiem, an independent scientific information platform dedicated to neurology for health care professionals and neuroscientists. These content quality guidelines are intended to be used by (1) content providers as a framework to meet content quality standards and (2) reviewers as a tool for analyzing and scoring quality of content.

Methods

Specific scientific criteria were designed using a 5-point scale to measure the quality of curated and original content published on the website: for Summaries, (1) source reliability and topic relevance for neurologists, (2) structure, and (3) scientific and didactic value; for Congress highlights, (1) relevance of congress selection, (2) congress coverage based on the original program, and (3) scientific and didactic value of individual abstracts; for Expert points of view and talks, (1) credibility (authorship) and topic relevance for neurologists, (2) scientific and didactic value, and (3) reliability (references) and format. The criteria were utilized on a monthly basis and endorsed by an independent scientific committee of widely recognized medical experts in neurology.

Results

Summary content quality for the 3 domains (reliability and relevance, structure, and scientific and didactic value) increased in the second month after the implementation of the guidelines. The domain scientific and didactic value had a mean score of 8.20/10. Scores for the domains reliability and relevance (8-9/10) and structure (45-55/60) showed that the maintenance of these 2 quality items over time was more challenging. Talks (either in the format of interviews or slide deck–supported scientific presentations) and expert point of view demonstrated high quality after the implementation of the content quality guidelines that was maintained over time (15-25/25).

Conclusions

Our findings support that content quality guidelines provide both (1) a reliable framework for generating independent high-quality content that addresses the educational needs of neurologists and (2) are an objective evaluation tool for improving and maintaining scientific quality level. The use of these criteria and this scoring system could serve as a standard and reference to build an editorial strategy and review process for any medical news or platforms.

Active Annotation in Evaluating the Credibility of Web-Based Medical Information: Guidelines for Creating Training Data Sets for Machine Learning

Background

The spread of false medical information on the web is rapidly accelerating. Establishing the credibility of web-based medical information has become a pressing necessity. Machine learning offers a solution that, when properly deployed, can be an effective tool in fighting medical misinformation on the web.

Objective

The aim of this study is to present a comprehensive framework for designing and curating machine learning training data sets for web-based medical information credibility assessment. We show how to construct the annotation process. Our main objective is to support researchers from the medical and computer science communities. We offer guidelines on the preparation of data sets for machine learning models that can fight medical misinformation.

Methods

We begin by providing the annotation protocol for medical experts involved in medical sentence credibility evaluation. The protocol is based on a qualitative study of our experimental data. To address the problem of insufficient initial labels, we propose a preprocessing pipeline for the batch of sentences to be assessed. It consists of representation learning, clustering, and reranking. We call this process active annotation.

Results

We collected more than 10,000 annotations of statements related to selected medical subjects (psychiatry, cholesterol, autism, antibiotics, vaccines, steroids, birth methods, and food allergy testing) for less than US $7000 by employing 9 highly qualified annotators (certified medical professionals), and we release this data set to the general public. We developed an active annotation framework for more efficient annotation of noncredible medical statements. The application of qualitative analysis resulted in a better annotation protocol for our future efforts in data set creation.

Conclusions

The results of the qualitative analysis support our claims of the efficacy of the presented method.