Detecting Medical Misinformation on Social Media Using Multimodal Deep Learning

Multimodal analysis of disinformation and misinformation

The use of disinformation and misinformation campaigns in the media has attracted much attention from academics and policy-makers. Multimodal analysis or the analysis of two or more semiotic systems-language, gestures, images, sounds, among others-in their interrelation and interaction is essential to understanding dis-/misinformation efforts because most human communication goes beyond just words. There is a confluence of many disciplines (e.g. computer science, linguistics, political science, communication studies) that are developing methods and analytical models of multimodal communication. This literature review brings research strands from these disciplines together, providing a map of the multi- and interdisciplinary landscape for multimodal analysis of dis-/misinformation. It records the substantial growth starting from the second quarter of 2020-the start of the COVID-19 epidemic in Western Europe-in the number of studies on multimodal dis-/misinformation coming from the field of computer science. The review examines that category of studies in more detail. Finally, the review identifies gaps in multimodal research on dis-/misinformation and suggests ways to bridge these gaps including future cross-disciplinary research directions. Our review provides scholars from different disciplines working on dis-/misinformation with a much needed bird's-eye view of the rapidly emerging research of multimodal dis-/misinformation.

Dung Beetle Optimization with Deep Feature Fusion Model for Lung Cancer Detection and Classification

Lung cancer is the main cause of cancer deaths all over the world. An important reason for these deaths was late analysis and worse prediction. With the accelerated improvement of deep learning (DL) approaches, DL can be effectively and widely executed for several real-world applications in healthcare systems, like medical image interpretation and disease analysis. Medical imaging devices can be vital in primary-stage lung tumor analysis and the observation of lung tumors from the treatment. Many medical imaging modalities like computed tomography (CT), chest X-ray (CXR), molecular imaging, magnetic resonance imaging (MRI), and positron emission tomography (PET) systems are widely analyzed for lung cancer detection. This article presents a new dung beetle optimization modified deep feature fusion model for lung cancer detection and classification (DBOMDFF-LCC) technique. The presented DBOMDFF-LCC technique mainly depends upon the feature fusion and hyperparameter tuning process. To accomplish this, the DBOMDFF-LCC technique uses a feature fusion process comprising three DL models, namely residual network (ResNet), densely connected network (DenseNet), and Inception-ResNet-v2. Furthermore, the DBO approach was employed for the optimum hyperparameter selection of three DL approaches. For lung cancer detection purposes, the DBOMDFF-LCC system utilizes a long short-term memory (LSTM) approach. The simulation result analysis of the DBOMDFF-LCC technique of the medical dataset is investigated using different evaluation metrics. The extensive comparative results highlighted the betterment of the DBOMDFF-LCC technique of lung cancer classification.

Strategies to Address COVID-19 Vaccine and Pregnancy Myths

ABSTRACT

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) poses risks to pregnant women and their infants. The spread of misinformation about COVID-19 vaccination is a barrier to optimizing vaccination rates among women of childbearing age. We conducted an environmental scan to identify misinformation about COVID-19 vaccination, pregnancy, and fertility, and a review to identify evidence to refute misinformation and strategies to correct and prevent the spread of misinformation. Seven identified themes of misinformation are: the vaccine causes female infertility; can cause miscarriage; and can decrease male fertility; mRNA vaccines attack the placenta; pregnant and breastfeeding persons should not get the vaccine; the vaccine can change menstrual cycles; and vaccinated people can spread infertility symptoms to unvaccinated people. Strategies that can be implemented by social media platforms to help prevent misinformation spread and correct existing health misinformation include improving information regulation by modifying community standards, implementing surveillance algorithms, and applying warning labels to potentially misleading posts. Health services organizations and clinicians can implement health misinformation policies, directly recommend vaccinations, provide credible explanations and resources to debunk misinformation, educate patients and populations on spotting misinformation, and apply effective communication strategies. More research is needed to assess longer-term effects of vaccination among women of childbearing age to strengthen the defense against misinformation and to evaluate strategies that aim to prevent and correct misinformation spread about COVID-19 vaccinations.

Forecasting User Interests Through Topic Tag Predictions in Online Health Communities

The increasing reliance on online communities for healthcare information by patients and caregivers has led to the increase in the spread of misinformation, or subjective, anecdotal and inaccurate or non-specific recommendations, which, if acted on, could cause serious harm to the patients. Hence, there is an urgent need to connect users with accurate and tailored health information in a timely manner to prevent such harm. This article proposes an innovative approach to suggesting reliable information to participants in online communities as they move through different stages in their disease or treatment. We hypothesize that patients with similar histories of disease progression or course of treatment would have similar information needs at comparable stages. Specifically, we pose the problem of predicting topic tags or keywords that describe the future information needs of users based on their profiles, traces of their online interactions within the community (past posts, replies) and the profiles and traces of online interactions of other users with similar profiles and similar traces of past interaction with the target users. The result is a variant of the collaborative information filtering or recommendation system tailored to the needs of users of online health communities. We report results of our experiments on two unique datasets from two different social media platforms which demonstrates the superiority of the proposed approach over the state of the art baselines with respect to accurate and timely prediction of topic tags (and hence information sources of interest).

Automatic detection of health misinformation: a systematic review

The spread of health misinformation has the potential to cause serious harm to public health, from leading to vaccine hesitancy to adoption of unproven disease treatments. In addition, it could have other effects on society such as an increase in hate speech towards ethnic groups or medical experts. To counteract the sheer amount of misinformation, there is a need to use automatic detection methods. In this paper we conduct a systematic review of the computer science literature exploring text mining techniques and machine learning methods to detect health misinformation. To organize the reviewed papers, we propose a taxonomy, examine publicly available datasets, and conduct a content-based analysis to investigate analogies and differences among Covid-19 datasets and datasets related to other health domains. Finally, we describe open challenges and conclude with future directions.

Knowledge Graphs: Opportunities and Challenges

With the explosive growth of artificial intelligence (AI) and big data, it has become vitally important to organize and represent the enormous volume of knowledge appropriately. As graph data, knowledge graphs accumulate and convey knowledge of the real world. It has been well-recognized that knowledge graphs effectively represent complex information; hence, they rapidly gain the attention of academia and industry in recent years. Thus to develop a deeper understanding of knowledge graphs, this paper presents a systematic overview of this field. Specifically, we focus on the opportunities and challenges of knowledge graphs. We first review the opportunities of knowledge graphs in terms of two aspects: (1) AI systems built upon knowledge graphs; (2) potential application fields of knowledge graphs. Then, we thoroughly discuss severe technical challenges in this field, such as knowledge graph embeddings, knowledge acquisition, knowledge graph completion, knowledge fusion, and knowledge reasoning. We expect that this survey will shed new light on future research and the development of knowledge graphs.

Visualizing the Interpretation of a Criteria-Driven System That Automatically Evaluates the Quality of Health News: Exploratory Study of 2 Approaches

BACKGROUND

Machine learning techniques have been shown to be efficient in identifying health misinformation, but the results may not be trusted unless they can be justified in a way that is understandable.

OBJECTIVE

This study aimed to provide a new criteria-based system to assess and justify health news quality. Using a subset of an existing set of criteria, this study compared the feasibility of 2 alternative methods for adding interpretability. Both methods used classification and highlighting to visualize sentence-level evidence.

METHODS

A total of 3 out of 10 well-established criteria were chosen for experimentation, namely whether the health news discussed the costs of the intervention (the cost criterion), explained or quantified the harms of the intervention (the harm criterion), and identified the conflicts of interest (the conflict criterion). The first step of the experiment was to automate the evaluation of the 3 criteria by developing a sentence-level classifier. We tested Logistic Regression, Naive Bayes, Support Vector Machine, and Random Forest algorithms. Next, we compared the 2 visualization approaches. For the first approach, we calculated word feature weights, which explained how classification models distill keywords that contribute to the prediction; then, using the local interpretable model-agnostic explanation framework, we selected keywords associated with the classified criterion at the document level; and finally, the system selected and highlighted sentences with keywords. For the second approach, we extracted sentences that provided evidence to support the evaluation result from 100 health news articles; based on these results, we trained a typology classification model at the sentence level; and then, the system highlighted a positive sentence instance for the result justification. The number of sentences to highlight was determined by a preset threshold empirically determined using the average accuracy.

RESULTS

The automatic evaluation of health news on the cost, harm, and conflict criteria achieved average area under the curve scores of 0.88, 0.76, and 0.73, respectively, after 50 repetitions of 10-fold cross-validation. We found that both approaches could successfully visualize the interpretation of the system but that the performance of the 2 approaches varied by criterion and highlighting the accuracy decreased as the number of highlighted sentences increased. When the threshold accuracy was ≥75%, this resulted in a visualization with a variable length ranging from 1 to 6 sentences.

CONCLUSIONS

We provided 2 approaches to interpret criteria-based health news evaluation models tested on 3 criteria. This method incorporated rule-based and statistical machine learning approaches. The results suggested that one might visually interpret an automatic criterion-based health news quality evaluation successfully using either approach; however, larger differences may arise when multiple quality-related criteria are considered. This study can increase public trust in computerized health information evaluation.

A Novel Hybrid Multi-Modal Deep Learning for Detecting Hashtag Incongruity on Social Media

Hashtags have been an integral element of social media platforms over the years and are widely used by users to promote, organize and connect users. Despite the intensive use of hashtags, there is no basis for using congruous tags, which causes the creation of many unrelated contents in hashtag searches. The presence of mismatched content in the hashtag creates many problems for individuals and brands. Although several methods have been presented to solve the problem by recommending hashtags based on the users' interest, the detection and analysis of the characteristics of these repetitive contents with irrelevant hashtags have rarely been addressed. To this end, we propose a novel hybrid deep learning hashtag incongruity detection by fusing visual and textual modality. We fine-tune BERT and ResNet50 pre-trained models to encode textual and visual information to encode textual and visual data simultaneously. We further attempt to show the capability of logo detection and face recognition in discriminating images. To extract faces, we introduce a pipeline that ranks faces based on the number of times they appear on Instagram accounts using face clustering. Moreover, we conduct our analysis and experiments on a dataset of Instagram posts that we collect from hashtags related to brands and celebrities. Unlike the existing works, we analyze these contents from both content and user perspectives and show a significant difference between data. In light of our results, we show that our multimodal model outperforms other models and the effectiveness of object detection in detecting mismatched information.

Improving medical experts' efficiency of misinformation detection: an exploratory study

Fighting medical disinformation in the era of the pandemic is an increasingly important problem. Today, automatic systems for assessing the credibility of medical information do not offer sufficient precision, so human supervision and the involvement of medical expert annotators are required. Our work aims to optimize the utilization of medical experts' time. We also equip them with tools for semi-automatic initial verification of the credibility of the annotated content. We introduce a general framework for filtering medical statements that do not require manual evaluation by medical experts, thus focusing annotation efforts on non-credible medical statements. Our framework is based on the construction of filtering classifiers adapted to narrow thematic categories. This allows medical experts to fact-check and identify over two times more non-credible medical statements in a given time interval without applying any changes to the annotation flow. We verify our results across a broad spectrum of medical topic areas. We perform quantitative, as well as exploratory analysis on our output data. We also point out how those filtering classifiers can be modified to provide experts with different types of feedback without any loss of performance.

Media Data and Vaccine Hesitancy: Scoping Review

Background

Media studies are important for vaccine hesitancy research, as they analyze how the media shapes risk perceptions and vaccine uptake. Despite the growth in studies in this field owing to advances in computing and language processing and an expanding social media landscape, no study has consolidated the methodological approaches used to study vaccine hesitancy. Synthesizing this information can better structure and set a precedent for this growing subfield of digital epidemiology.

Objective

This review aimed to identify and illustrate the media platforms and methods used to study vaccine hesitancy and how they build or contribute to the study of the media's influence on vaccine hesitancy and public health.

Methods

This study followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. A search was conducted on PubMed and Scopus for any studies that used media data (social media or traditional media), had an outcome related to vaccine sentiment (opinion, uptake, hesitancy, acceptance, or stance), were written in English, and were published after 2010. Studies were screened by only 1 reviewer and extracted for media platform, analysis method, the theoretical models used, and outcomes.

Results

In total, 125 studies were included, of which 71 (56.8%) used traditional research methods and 54 (43.2%) used computational methods. Of the traditional methods, most used content analysis (43/71, 61%) and sentiment analysis (21/71, 30%) to analyze the texts. The most common platforms were newspapers, print media, and web-based news. The computational methods mostly used sentiment analysis (31/54, 57%), topic modeling (18/54, 33%), and network analysis (17/54, 31%). Fewer studies used projections (2/54, 4%) and feature extraction (1/54, 2%). The most common platforms were Twitter and Facebook. Theoretically, most studies were weak. The following five major categories of studies arose: antivaccination themes centered on the distrust of institutions, civil liberties, misinformation, conspiracy theories, and vaccine-specific concerns; provaccination themes centered on ensuring vaccine safety using scientific literature; framing being important and health professionals and personal stories having the largest impact on shaping vaccine opinion; the coverage of vaccination-related data mostly identifying negative vaccine content and revealing deeply fractured vaccine communities and echo chambers; and the public reacting to and focusing on certain signals-in particular cases, deaths, and scandals-which suggests a more volatile period for the spread of information.

Conclusions

The heterogeneity in the use of media to study vaccines can be better consolidated through theoretical grounding. Areas of suggested research include understanding how trust in institutions is associated with vaccine uptake, how misinformation and information signaling influence vaccine uptake, and the evaluation of government communications on vaccine rollouts and vaccine-related events. The review ends with a statement that media data analyses, though groundbreaking in approach, should supplement-not supplant-current practices in public health research.

Research on Image Segmentation Algorithm Based on Multimodal Hierarchical Attention Mechanism and Genetic Neural Network

Multimodal tasks based on attention mechanism and language face numerous problems. Based on multimodal hierarchical attention mechanism and genetic neural network, this paper studies the application of image segmentation algorithm in data completion and 3D scene reconstruction. The algorithm refers to the process of concentrating attention that humans subjectively pay attention to and calculates the difference between each pixel in the genetic neural network test image in the color space and the average value of the target image, which solves the problem of static feature maps and dynamic feature maps of image sequences. In addition, in view of the problem that the number of attention enhancement feature extraction modules is too large and the parameters are too large, the recursive mechanism is used as the feature extraction branch, and new model parameters are not added when the network depth is increased. The simulation results show that the accuracy of the improved image saliency detection algorithm based on the attention mechanism reaches 89.7%, and the difference between the average value of the single-point pixel and the target image is reduced to 0.132, which further promotes the practicability and reliability of the image segmentation model.

COVID-19 vaccine hesitancy: a social media analysis using deep learning

Hesitant attitudes have been a significant issue since the development of the first vaccines-the WHO sees them as one of the most critical global health threats. The increasing use of social media to spread questionable information about vaccination strongly impacts the population's decision to get vaccinated. Developing text classification methods that can identify hesitant messages on social media could be useful for health campaigns in their efforts to address negative influences from social media platforms and provide reliable information to support their strategies against hesitant-vaccination sentiments. This study aims to evaluate the performance of different machine learning models and deep learning methods in identifying vaccine-hesitant tweets that are being published during the COVID-19 pandemic. Our concluding remarks are that Long Short-Term Memory and Recurrent Neural Network models have outperformed traditional machine learning models on detecting vaccine-hesitant messages in social media, with an accuracy rate of 86% against 83%.

Identifying False Human Papillomavirus (HPV) Vaccine Information and Corresponding Risk Perceptions From Twitter: Advanced Predictive Models

Background

The vaccination uptake rates of the human papillomavirus (HPV) vaccine remain low despite the fact that the effectiveness of HPV vaccines has been established for more than a decade. Vaccine hesitancy is in part due to false information about HPV vaccines on social media. Combating false HPV vaccine information is a reasonable step to addressing vaccine hesitancy.

Objective

Given the substantial harm of false HPV vaccine information, there is an urgent need to identify false social media messages before it goes viral. The goal of the study is to develop a systematic and generalizable approach to identifying false HPV vaccine information on social media.

Methods

This study used machine learning and natural language processing to develop a series of classification models and causality mining methods to identify and examine true and false HPV vaccine–related information on Twitter.

Results

We found that the convolutional neural network model outperformed all other models in identifying tweets containing false HPV vaccine–related information (F score=91.95). We also developed completely unsupervised causality mining models to identify HPV vaccine candidate effects for capturing risk perceptions of HPV vaccines. Furthermore, we found that false information contained mostly loss-framed messages focusing on the potential risk of vaccines covering a variety of topics using more diverse vocabulary, while true information contained both gain- and loss-framed messages focusing on the effectiveness of vaccines covering fewer topics using relatively limited vocabulary.

Conclusions

Our research demonstrated the feasibility and effectiveness of using predictive models to identify false HPV vaccine information and its risk perceptions on social media.

Using Machine Learning to Compare Provaccine and Antivaccine Discourse Among the Public on Social Media: Algorithm Development Study

BACKGROUND

Despite numerous counteracting efforts, antivaccine content linked to delays and refusals to vaccinate has grown persistently on social media, while only a few provaccine campaigns have succeeded in engaging with or persuading the public to accept immunization. Many prior studies have associated the diversity of topics discussed by antivaccine advocates with the public's higher engagement with such content. Nonetheless, a comprehensive comparison of discursive topics in pro- and antivaccine content in the engagement-persuasion spectrum remains unexplored.

OBJECTIVE

We aimed to compare discursive topics chosen by pro- and antivaccine advocates in their attempts to influence the public to accept or reject immunization in the engagement-persuasion spectrum. Our overall objective was pursued through three specific aims as follows: (1) we classified vaccine-related tweets into provaccine, antivaccine, and neutral categories; (2) we extracted and visualized discursive topics from these tweets to explain disparities in engagement between pro- and antivaccine content; and (3) we identified how those topics frame vaccines using Entman's four framing dimensions.

METHODS

We adopted a multimethod approach to analyze discursive topics in the vaccine debate on public social media sites. Our approach combined (1) large-scale balanced data collection from a public social media site (ie, 39,962 tweets from Twitter); (2) the development of a supervised classification algorithm for categorizing tweets into provaccine, antivaccine, and neutral groups; (3) the application of an unsupervised clustering algorithm for identifying prominent topics discussed on both sides; and (4) a multistep qualitative content analysis for identifying the prominent discursive topics and how vaccines are framed in these topics. In so doing, we alleviated methodological challenges that have hindered previous analyses of pro- and antivaccine discursive topics.

RESULTS

Our results indicated that antivaccine topics have greater intertopic distinctiveness (ie, the degree to which discursive topics are distinct from one another) than their provaccine counterparts (t₁₂₂=2.30, P=.02). In addition, while antivaccine advocates use all four message frames known to make narratives persuasive and influential, provaccine advocates have neglected having a clear problem statement.

CONCLUSIONS

Based on our results, we attribute higher engagement among antivaccine advocates to the distinctiveness of the topics they discuss, and we ascribe the influence of the vaccine debate on uptake rates to the comprehensiveness of the message frames. These results show the urgency of developing clear problem statements for provaccine content to counteract the negative impact of antivaccine content on uptake rates.