Text mining for social science - The state and the future of computational text analysis in sociology

A systematic evaluation of text mining methods for short texts: Mapping individuals' internal states from online posts

Short texts generated by individuals in online environments can provide social and behavioral scientists with rich insights into these individuals' internal states. Trained manual coders can reliably interpret expressions of such internal states in text. However, manual coding imposes restrictions on the number of texts that can be analyzed, limiting our ability to extract insights from large-scale textual data. We evaluate the performance of several automatic text analysis methods in approximating trained human coders' evaluations across four coding tasks encompassing expressions of motives, norms, emotions, and stances. Our findings suggest that commonly used dictionaries, although performing well in identifying infrequent categories, generate false positives too frequently compared to other methods. We show that large language models trained on manually coded data yield the highest performance across all case studies. However, there are also instances where simpler methods show almost equal performance. Additionally, we evaluate the effectiveness of cutting-edge generative language models like GPT-4 in coding texts for internal states with the help of short instructions (so-called zero-shot classification). While promising, these models fall short of the performance of models trained on manually analyzed data. We discuss the strengths and weaknesses of various models and explore the trade-offs between model complexity and performance in different applications. Our work informs social and behavioral scientists of the challenges associated with text mining of large textual datasets, while providing best-practice recommendations.

"Is a game really a reason for people to die?" Sentiment and thematic analysis of Twitter-based discourse on Indonesia soccer stampede

This study examined discourses related to an Indonesian soccer stadium stampede on 1st October 2022 using comments posted on Twitter. We conducted a lexicon-based sentiment analysis to identify the sentiments and emotions expressed in tweets and performed structural topic modeling to identify latent themes in the discourse. The majority of tweets (87.8%) expressed negative sentiments, while 8.2% and 4.0% of tweets expressed positive and neutral sentiments, respectively. The most common emotion expressed was fear (29.3%), followed by sadness and anger. Of the 19 themes identified, "Deaths and mortality" was the most prominent (15.1%), followed by "family impact". The negative stampede discourse was related to public concerns such as "vigil" and "calls for bans and suspension," while positive discourse focused more on the impact of the stampede. Public health institutions can leverage the volume and rapidity of social media to improve disaster prevention strategies.

Examining ethno-racial attitudes of the public in Twitter discourses related to the United States Supreme Court Dobbs vs. Jackson Women's Health Organization ruling: A machine learning approach

Background

The decision of the US Supreme Court to repeal Roe vs. Wade sparked significant media attention. Although primarily related to abortion, opinions are divided about how this decision would impact disparities, especially for Black, Indigenous, and people of color. We used advanced natural language processing (NLP) techniques to examine ethno-racial contents in Twitter discourses related to the overturn of Roe vs. Wade.

Methods

We screened approximately 3 million tweets posted to Roe vs. Wade discussions and identified unique tweets in English-language that had mentions related to race, ethnicity, and racism posted between June 24 and July 10, 2022. We performed lexicon-based sentiment analysis to identify sentiment polarity and the emotions expressed in the Twitter discourse and conducted structural topic modeling to identify and examine latent themes.

Results

Of the tweets retrieved, 0.7% (n = 23,044) had mentions related to race, ethnicity, and racism. The overall sentiment polarity was negative (mean = -0.41, SD = 1.48). Approximately 60.0% (n = 12,092) expressed negative sentiments, while 39.0% (n = 81,45) expressed positive sentiments, and 3.0% (n = 619) expressed neutral sentiments. There were 20 latent themes which emerged from the topic model. The predominant topics in the discourses were related to "racial resentment" (topic 2, 11.3%), "human rights" (topic 2, 7.9%), and "socioeconomic disadvantage" (topic 16, 7.4%).

Conclusions

Our study demonstrates wide ranging ethno-racial concerns following the reversal of Roe and supports the need for active surveillance of racial and ethnic disparities in abortion access in the post-Roe era.