They May not Work! An Evaluation of Eleven Sentiment Analysis Tools on Seven Social Media Datasets

Sentiment analysis of U.S. News & World Report Best Children's Hospital urology rankings: A difference in positivity between the public and academic worlds

INTRODUCTION

Initiated in 2009, the U.S. News & World Report (USNWR) pediatric urology rankings aim to guide patients and families towards high-quality urologic care. Despite this, the pediatric urology community remains divided, with significant debate over the rankings' accuracy, utility, and potential for misleading information. While some professionals argue for a collective opt-out from these rankings, citing these concerns, others highlight their positive impact on patient care, hospital benchmarking, and financial support.

OBJECTIVE

Recognizing the lack of formal evaluation on how these rankings are viewed beyond the pediatric urology community, this research endeavors to fill the gap through sentiment analysis of public news articles and academic publications.

STUDY DESIGN

We captured news articles from Google News and academic papers from Ovid Medline and Embase, focusing specifically on content related to the USNWR pediatric urology rankings from 2009 to 2023. Sentiment analysis was conducted using the Valence Aware Dictionary and Sentiment Reasoner (VADER) package on both news and academic texts, aiming to capture the overall sentiment through a compound score derived from the presence of sentiment-laden words. Sensitivity analysis was performed using TextBlob Pattern Analyzer tool.

RESULTS

The analysis revealed a significant divergence in sentiment between news articles and academic literature. News articles exhibited a predominantly positive sentiment, with an average compound score of 0.681, suggesting a general approval or celebration of the rankings in the public sphere. Conversely, academic literature showed a more moderate sentiment, with an average score of 0.534, indicating a nuanced perspective that includes both positive views and critical reflections on the rankings. Sensitivity analysis confirmed this observation (Figure).

DISCUSSION

This difference may reflect the distinct nature of news media and academic discourse. While news outlets may prioritize celebratory narratives that align with public interest and institutional pride, academic discussions tend to offer a balanced view that critically assesses both the merits and limitations of the rankings. This discrepancy underscores the complexity of interpreting and acting upon the rankings within the pediatric urology community.

CONCLUSION

While the USNWR pediatric urology rankings are generally received positively by the public, as reflected in news media, the academic community presents a more reserved sentiment. These findings suggest the need for ongoing dialogue and research to understand the implications of these rankings fully. It also calls for a strategic approach to address the concerns and perceptions of healthcare professionals, aiming to leverage the rankings in a way that truly benefits patient care and informed decision-making.

Measuring the impact of scientific publications and publication extenders: examples of novel approaches

Different stakeholders, such as authors, research institutions, and healthcare professionals (HCPs) may determine the impact of peer-reviewed publications in different ways. Commonly-used measures of research impact, such as the Journal Impact Factor or the H-index, are not designed to evaluate the impact of individual articles. They are heavily dependent on citations, and therefore only measure impact of the overall journal or researcher respectively, taking months or years to accrue. The past decade has seen the development of article-level metrics (ALMs), that measure the online attention received by an individual publication in contexts including social media platforms, news media, citation activity, and policy and patent citations. These new tools can complement traditional bibliometric data and provide a more holistic evaluation of the impact of a publication. This commentary discusses the need for ALMs, and summarizes several examples - PlumX Metrics, Altmetric, the Better Article Metrics score, the EMPIRE Index, and scite. We also discuss how metrics may be used to evaluate the value of "publication extenders" - educational microcontent such as animations, videos and plain-language summaries that are often hosted on HCP education platforms. Publication extenders adapt a publication's key data to audience needs and thereby extend a publication's reach. These new approaches have the potential to address the limitations of traditional metrics, but the diversity of new metrics requires that users have a keen understanding of which forms of impact are relevant to a specific publication and select and monitor ALMs accordingly.

A Comparison of ChatGPT and Fine-Tuned Open Pre-Trained Transformers (OPT) Against Widely Used Sentiment Analysis Tools: Sentiment Analysis of COVID-19 Survey Data

BACKGROUND

Health care providers and health-related researchers face significant challenges when applying sentiment analysis tools to health-related free-text survey data. Most state-of-the-art applications were developed in domains such as social media, and their performance in the health care context remains relatively unknown. Moreover, existing studies indicate that these tools often lack accuracy and produce inconsistent results.

OBJECTIVE

This study aims to address the lack of comparative analysis on sentiment analysis tools applied to health-related free-text survey data in the context of COVID-19. The objective was to automatically predict sentence sentiment for 2 independent COVID-19 survey data sets from the National Institutes of Health and Stanford University.

METHODS

Gold standard labels were created for a subset of each data set using a panel of human raters. We compared 8 state-of-the-art sentiment analysis tools on both data sets to evaluate variability and disagreement across tools. In addition, few-shot learning was explored by fine-tuning Open Pre-Trained Transformers (OPT; a large language model [LLM] with publicly available weights) using a small annotated subset and zero-shot learning using ChatGPT (an LLM without available weights).

RESULTS

The comparison of sentiment analysis tools revealed high variability and disagreement across the evaluated tools when applied to health-related survey data. OPT and ChatGPT demonstrated superior performance, outperforming all other sentiment analysis tools. Moreover, ChatGPT outperformed OPT, exhibited higher accuracy by 6% and higher F-measure by 4% to 7%.

CONCLUSIONS

This study demonstrates the effectiveness of LLMs, particularly the few-shot learning and zero-shot learning approaches, in the sentiment analysis of health-related survey data. These results have implications for saving human labor and improving efficiency in sentiment analysis tasks, contributing to advancements in the field of automated sentiment analysis.

Users' Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts

BACKGROUND

Within a few months, the COVID-19 pandemic had spread to many countries and had been a real challenge for health systems all around the world. This unprecedented crisis has led to a surge of online discussions about potential cures for the disease. Among them, vaccines have been at the heart of the debates and have faced lack of confidence before marketing in France.

OBJECTIVE

This study aims to identify and investigate the opinions of French Twitter users on the announced vaccines against COVID-19 through sentiment analysis.

METHODS

This study was conducted in 2 phases. First, we filtered a collection of tweets related to COVID-19 available on Twitter from February 2020 to August 2020 with a set of keywords associated with vaccine mistrust using word embeddings. Second, we performed sentiment analysis using deep learning to identify the characteristics of vaccine mistrust. The model was trained on a hand-labeled subset of 4548 tweets.

RESULTS

A set of 69 relevant keywords were identified as the semantic concept of the word "vaccin" (vaccine in French) and focused mainly on conspiracies, pharmaceutical companies, and alternative treatments. Those keywords enabled us to extract nearly 350,000 tweets in French. The sentiment analysis model achieved 0.75 accuracy. The model then predicted 16% of positive tweets, 41% of negative tweets, and 43% of neutral tweets. This allowed us to explore the semantic concepts of positive and negative tweets and to plot the trends of each sentiment. The main negative rhetoric identified from users' tweets was that vaccines are perceived as having a political purpose and that COVID-19 is a commercial argument for the pharmaceutical companies.

CONCLUSIONS

Twitter might be a useful tool to investigate the arguments for vaccine mistrust because it unveils political criticism contrasting with the usual concerns on adverse drug reactions. As the opposition rhetoric is more consistent and more widely spread than the positive rhetoric, we believe that this research provides effective tools to help health authorities better characterize the risk of vaccine mistrust.