Information quality of videos related to Helicobacter pylori infection on TikTok: Cross-sectional study

Analyzing dissemination, quality, and reliability of Chinese brain tumor-related short videos on TikTok and Bilibili: a cross-sectional study

Background

As the Internet becomes an increasingly vital source of medical information, the quality and reliability of brain tumor-related short videos on platforms such as TikTok and Bilibili have not been adequately evaluated. Therefore, this study aims to assess these aspects and explore the factors influencing the dissemination of such videos.

Methods

A cross-sectional analysis was conducted on the top 100 brain tumor-related short videos from TikTok and Bilibili. The videos were evaluated using the Global Quality Score and the DISCERN reliability instrument. An eXtreme Gradient Boosting algorithm was utilized to predict dissemination outcomes. The videos were also categorized by content type and uploader.

Results

TikTok videos scored relatively higher on both the Global Quality Score (median 2, interquartile range [2, 3] on TikTok vs. median 2, interquartile range [1, 2] on Bilibili, p = 1.51E-04) and the DISCERN reliability instrument (median 15, interquartile range [13, 18.25] on TikTok vs. 13.5, interquartile range [11, 16] on Bilibili, p = 1.66E-04). Subgroup analysis revealed that videos uploaded by professional individuals and institutions had higher quality and reliability compared to those uploaded by non-professional entities. Videos focusing on disease knowledge exhibited the highest quality and reliability compared to other content types. The number of followers emerged as the most important variable in our dissemination prediction model.

Conclusion

The overall quality and reliability of brain tumor-related short videos on TikTok and Bilibili were unsatisfactory and did not significantly influence video dissemination. Future research should expand the scope to better understand the factors driving the dissemination of medical-themed videos.

Exploring the capacities of ChatGPT: A comprehensive evaluation of its accuracy and repeatability in addressing helicobacter pylori-related queries

BACKGROUND

Educational initiatives on Helicobacter pylori (H. pylori) constitute a highly effective approach for preventing its infection and establishing standardized protocols for its eradication. ChatGPT, a large language model, is a potentially patient-friendly online tool capable of providing health-related knowledge. This study aims to assess the accuracy and repeatability of ChatGPT in responding to questions related to H. pylori.

MATERIALS AND METHODS

Twenty-one common questions about H. pylori were collected and categorized into four domains: basic knowledge, diagnosis, treatment, and prevention. ChatGPT was utilized to individually answer the aforementioned 21 questions. Its responses were independently assessed by two experts on H. pylori. Questions with divergent ratings were resolved by a third reviewer. Cohen's kappa coefficient was calculated to assess the consistency between the scores of the two reviewers.

RESULTS

The responses of ChatGPT on H. pylori-related questions were generally satisfactory, with 61.9% marked as "completely correct" and 33.33% as "correct but inadequate." The repeatability of the responses of ChatGPT to H. pylori-related questions was 95.23%. Among the responses, those related to prevention (comprehensive: 75%) had the best response, followed by those on treatment (comprehensive: 66.7%), basic knowledge (comprehensive: 60%), and diagnosis (comprehensive: 50%). In the "treatment" domain, 16.6% of the ChatGPT responses were categorized as "mixed with correct or incorrect/outdated data." However, ChatGPT still lacks relevant knowledge regarding H. pylori resistance and the use of sensitive antibiotics.

CONCLUSIONS

ChatGPT can provide correct answers to the majority of H. pylori-related queries. It exhibited good reproducibility and delivered responses that were easily comprehensible to patients. Further enhancement of real-time information updates and correction of inaccurate information will make ChatGPT an essential auxiliary tool for providing accurate H. pylori-related health information to patients.

The status quo of short videos as a health information source of Helicobacter pylori: a cross-sectional study

Background

Health education about Helicobacter pylori (H. pylori) is one of the most effective methods to prevent H. pylori infection and standardize H. pylori eradication treatment. Short videos enable people to absorb and remember information more easily and are an important source of health education. This study aimed to assess the information quality of H. pylori-related videos on Chinese short video-sharing platforms.

Methods

A total of 242 H. pylori-related videos from three Chinese short video-sharing platforms with the most users, TikTok, Bilibili, and Kwai, were retrieved. The Global Quality Score (GQS) and the modified DISCERN tool were used to assess the quality and content of videos, respectively. Additionally, comparative analyzes of videos based on different sources and common H. pylori issues were also conducted.

Results

The median GQS score and DISCERN score was 2 for H. pylori-related videos analyzed in this study. Non-gastroenterologists posted the most H. pylori-related videos (136/242, 56.2%). Videos from gastroenterologists (51/242, 21.0%) had the highest GQS and DISCERN scores, with a median of 3. Few videos had content on family-based H. pylori infection control and management (5.8%), whether all H. pylori-positive patients need to undergo eradication treatment (27.7%), and the adverse effects of H. pylori eradication therapy (16.1%).

Conclusion

Generally, the content and quality of the information in H. pylori-related videos were unsatisfactory, and the quality of the video correlated with the source of the video. Videos from gastroenterologists provided more correct guidance with higher-quality information on the prevention and treatment of H. pylori infection.

Comparative analysis of large language models in medical counseling: A focus on Helicobacter pylori infection

BACKGROUND

Large language models (LLMs) are promising medical counseling tools, but the reliability of responses remains unclear. We aimed to assess the feasibility of three popular LLMs as counseling tools for Helicobacter pylori infection in different counseling languages.

MATERIALS AND METHODS

This study was conducted between November 20 and December 1, 2023. Three large language models (ChatGPT 4.0 [LLM1], ChatGPT 3.5 [LLM2], and ERNIE Bot 4.0 [LLM3]) were input 15 H. pylori related questions each, once in English and once in Chinese. Each chat was conducted using the "New Chat" function to avoid bias from correlation interference. Responses were recorded and blindly assigned to three reviewers for scoring on three established Likert scales: accuracy (ranged 1-6 point), completeness (ranged 1-3 point), and comprehensibility (ranged 1-3 point). The acceptable thresholds for the scales were set at a minimum of 4, 2, and 2, respectively. Final various source and interlanguage comparisons were made.

RESULTS

The overall mean (SD) accuracy score was 4.80 (1.02), while 1.82 (0.78) for completeness score and 2.90 (0.36) for comprehensibility score. The acceptable proportions for the accuracy, completeness, and comprehensibility of the responses were 90%, 45.6%, and 100%, respectively. The acceptable proportion of overall completeness score for English responses was better than for Chinese responses (p = 0.034). For accuracy, the English responses of LLM3 were better than the Chinese responses (p = 0.0055). As for completeness, the English responses of LLM1 was better than the Chinese responses (p = 0.0257). For comprehensibility, the English responses of LLM1 was better than the Chinese responses (p = 0.0496). No differences were found between the various LLMs.

CONCLUSIONS

The LLMs responded satisfactorily to questions related to H. pylori infection. But further improving completeness and reliability, along with considering language nuances, is crucial for optimizing overall performance.