Using the Google™ Search Engine for Health Information: Is There a Problem? Case Study: Supplements for Cancer

Use of Generative AI for Improving Health Literacy in Reproductive Health: Case Study

BACKGROUND

Patients find technology tools to be more approachable for seeking sensitive health-related information, such as reproductive health information. The inventive conversational ability of artificial intelligence (AI) chatbots, such as ChatGPT (OpenAI Inc), offers a potential means for patients to effectively locate answers to their health-related questions digitally.

OBJECTIVE

A pilot study was conducted to compare the novel ChatGPT with the existing Google Search technology for their ability to offer accurate, effective, and current information regarding proceeding action after missing a dose of oral contraceptive pill.

METHODS

A sequence of 11 questions, mimicking a patient inquiring about the action to take after missing a dose of an oral contraceptive pill, were input into ChatGPT as a cascade, given the conversational ability of ChatGPT. The questions were input into 4 different ChatGPT accounts, with the account holders being of various demographics, to evaluate potential differences and biases in the responses given to different account holders. The leading question, "what should I do if I missed a day of my oral contraception birth control?" alone was then input into Google Search, given its nonconversational nature. The results from the ChatGPT questions and the Google Search results for the leading question were evaluated on their readability, accuracy, and effective delivery of information.

RESULTS

The ChatGPT results were determined to be at an overall higher-grade reading level, with a longer reading duration, less accurate, less current, and with a less effective delivery of information. In contrast, the Google Search resulting answer box and snippets were at a lower-grade reading level, shorter reading duration, more current, able to reference the origin of the information (transparent), and provided the information in various formats in addition to text.

CONCLUSIONS

ChatGPT has room for improvement in accuracy, transparency, recency, and reliability before it can equitably be implemented into health care information delivery and provide the potential benefits it poses. However, AI may be used as a tool for providers to educate their patients in preferred, creative, and efficient ways, such as using AI to generate accessible short educational videos from health care provider-vetted information. Larger studies representing a diverse group of users are needed.

An Artificial Intelligence Chatbot is an Accurate and Useful Online Patient Resource Prior to Total Knee Arthroplasty

BACKGROUND

Online information is a useful resource for patients seeking advice on their orthopaedic care. While traditional websites provide responses to specific frequently asked questions (FAQs), sophisticated artificial intelligence tools may be able to provide the same information to patients in a more accessible manner. Chat Generative Pretrained Transformer (ChatGPT) is a powerful artificial intelligence chatbot that has been shown to effectively draw on its large reserves of information in a conversational context with a user. The purpose of this study was to assess the accuracy and reliability of ChatGPT-generated responses to FAQs regarding total knee arthroplasty.

METHODS

We distributed a survey that challenged arthroplasty surgeons to identify which of the 2 responses to FAQs on our institution's website was human-written and which was generated by ChatGPT. All questions were total knee arthroplasty-related. The second portion of the survey investigated the potential to further leverage ChatGPT to assist with translation and accessibility as a means to better meet the needs of our diverse patient population.

RESULTS

Surgeons correctly identified the ChatGPT-generated responses 4 out of 10 times on average (range: 0 to 7). No consensus was reached on any of the responses to the FAQs. Additionally, over 90% of our surgeons strongly encouraged the use of ChatGPT to more effectively accommodate the diverse patient populations that seek information from our hospital's online resources.

CONCLUSIONS

ChatGPT provided accurate, reliable answers to our website's FAQs. Surgeons also agreed that ChatGPT's ability to provide targeted, language-specific responses to FAQs would be of benefit to our diverse patient population.

Educating patients on osteoporosis and bone health: Can "ChatGPT" provide high-quality content?

PURPOSE

The rise of artificial intelligence (AI) models like ChatGPT offers potential for varied applications, including patient education in healthcare. With gaps in osteoporosis and bone health knowledge and adherence to prevention and treatment, this study aims to evaluate the accuracy of ChatGPT in delivering evidence-based information related to osteoporosis.

METHODS

Twenty of the most common frequently asked questions (FAQs) related to osteoporosis were subcategorized into diagnosis, diagnostic method, risk factors, and treatment and prevention. These FAQs were sourced online and inputted into ChatGPT-3.5. Three orthopedic surgeons and one advanced practice provider who routinely treat patients with fragility fractures independently reviewed the ChatGPT-generated answers, grading them on a scale from 0 (harmful) to 4 (excellent). Mean response accuracy scores were calculated. To compare the variance of the means across the four categories, a one-way analysis of variance (ANOVA) was used.

RESULTS

ChatGPT displayed an overall mean accuracy score of 91%. Its responses were graded as "accurate requiring minimal clarification" or "excellent," with a mean response score ranging from 3.25 to 4. No answers were deemed inaccurate or harmful. No significant difference was observed in the means of responses across the defined categories.

CONCLUSION

ChatGPT-3.5 provided high-quality educational content. It showcased a high degree of accuracy in addressing osteoporosis-related questions, aligning closely with expert opinions and current literature, with structured and inclusive answers. However, while AI models can enhance patient information accessibility, they should be used as an adjunct rather than a substitute for human expertise and clinical judgment.

Assessment of internet-based information on statin therapy

AIMS

The use of statin therapy is deemed to be controversial by mainstream media. Patients increasingly source medical information from the internet, and the use of statins is no exception. This study aims to determine the quality and educational content of statin-focused information on the internet and YouTube.

METHODS AND RESULTS

'Statin' was searched on Google, Yahoo!, Bing, and YouTube. The first 50 results obtained from each search engine and the first 20 YouTube videos were screened by two assessors. Websites were assessed using the Flesch Reading Ease (FRE) score, University of Michigan Consumer Health Website Evaluation Checklist, and a customized scoring system evaluating statin-focused content for quality. Videos were scored using the Journal of the American Medical Association (JAMA) benchmark criteria, Global Quality Score (GQS), and the customized scoring system. Websites scored a median FRE score of 57.5 [interquartile range (IQR) 52.1-62.3], median Michigan score of 36 (IQR 32-41.5), and median content score of 5 (IQR 3.75-7). Good interobserver agreement was demonstrated [Michigan score interobserver coefficient correlation (ICC) = 0.968; content score ICC = 0.944]. Videos scored a median JAMA score of 2, median GQS score of 2.5, and median content score of 2.5. Good interobserver agreement was demonstrated (JAMA ICC = 0.746; GQS ICC = 0.874; content score ICC = 0.946).

CONCLUSION

Quality and readability of statin-focused online information are poor. Healthcare professionals should be aware of the limitations of the current available sources and design online resources that are accurate and patient-friendly.

Clear as Mud: Readability Scores in Cloacal Exstrophy Literature and Its Treatment

Purpose

This study examines the readability of online medical information regarding cloacal exstrophy (CE). We hypothesize that inappropriate levels of comprehension are required in these resources, leading to poor understanding and confusion amongst caregivers.

Methods

The Google and Bing search engines were used to search the terms "cloacal exstrophy" and "cloacal exstrophy treatment". The first 100 results for each were collected. Each webpage was analyzed for readability using four independent validated scoring systems: the Gunning-Fog index (GFI), SMOG grade (Simple Measure of Gobbledygook), Dale-Chall index (DCI), and the Flesch-Kincaid grade (FKG).

Results

Forty-seven unique webpages fit the inclusion criteria. Mean readability scores across all websites were GFI, 14.6; SMOG score, 10.8; DCI, 9.3; and FKG, 11.8, correlating to adjusted grade levels of college sophomore, 11th grade, college, and 11th grade, respectively. There were significant differences across all readability formulas. Non-profit websites were significantly less readable than institutional and commercial webpages (GFI p = 0.012, SMOG p = 0.018, DCI p = 0.021, FKG p = 0.0093).

Conclusion

Caregiver-directed health information regarding CE and its treatment available online is written at the 11th grade reading level or above. Online resources pertaining to CE must be simplified to be effective.

Are popular books about diet and health written based on scientific evidence? A comparison of citations between the USA and Japan

OBJECTIVES

To describe and compare the references cited in popular books about diet and health between the USA and Japan.

DESIGN

Books were selected based on their best-seller rankings in the diet and health category of online bookstores. We identified references throughout all pages of the books and examined the number of references, reference format (identifiable or not) and presence of specific types of references, such as systematic reviews of human research. We compared the characteristics of references between the two countries and examined related factors to citation.

SETTING

Cross-sectional study.

PARTICIPANTS

Books (n 100 in each country).

RESULTS

Among 100 books from each country, sixty-five US and sixty-six Japanese books had references. Forty-five US books cited more than 100 references, against only five Japanese books. The number of books that cited systematic reviews of human research differed between the USA (n 49) and Japan (n 9). Additionally, the number of books that provided identifiable information for all references was significantly higher in the USA (n 63) than in Japan (n 42). Books whose first authors have licences of medical doctors were more likely to cite references than those without in both countries.

CONCLUSIONS

Two-thirds of books about diet and health cited references in both the USA and Japan, but Japanese books cited fewer references and were less likely to cite systematic reviews and provide identifiable references than US books. Further research into the scientific reliability of information in books about diet and health is warranted.

Web-Based Content on Diet and Nutrition Written in Japanese: Infodemiology Study Based on Google Trends and Google Search

BACKGROUND

The increased availability of content of uncertain integrity obtained through the internet is a major concern. To date, however, there has been no comprehensive scrutiny of the fitness-for-purpose of web-based content on diet and nutrition.

OBJECTIVE

This cross-sectional study aims to describe diet- and nutrition-related web-based content written in Japanese, identified via a systematic extraction strategy using Google Trends and Google Search.

METHODS

We first identified keywords relevant for extracting web-based content (eg, blogs) on diet and nutrition written in Japanese using Google Trends. This process included identification of 638 seed terms, identification of approximately 1500 pairs of related queries (top) and search terms, the top 10% of which were extracted to identify 160 relevant pairs of related queries (top) and search terms, and identification of 107 keywords for search. We then extracted relevant web-based content using Google Search.

RESULTS

The content (N=1703) examined here was extracted following a search based on 107 keywords. The most common themes included food and beverages (390/1703, 22.9%), weight management (366/1703, 21.49%), health benefits (261/1703, 15.33%), and healthy eating (235/1703, 13.8%). The main disseminators were information technology companies and mass media (474/1703, 27.83%), food manufacturers (246/1703, 14.45%), other (236/1703, 13.86%), and medical institutions (214/1703, 12.57%). Less than half of the content (790/1703, 46.39%) clearly indicated the involvement of editors or writers. More than half of the content (983/1703, 57.72%) was accompanied by one or more types of advertisement. The proportion of content with any type of citation reference was 40.05% (682/1703). The themes and disseminators of content were significantly associated with the involvement of editors or writers, accompaniment with advertisement, and citation of reference. In particular, content focusing on weight management was more likely to clearly indicate the involvement of editors or writers (212/366, 57.9%) and to be accompanied by advertisement (273/366, 74.6%), but less likely to have references cited (128/366, 35%). Content from medical institutions was less likely to have citation references (62/214, 29%).

CONCLUSIONS

This study highlights concerns regarding the authorship, conflicts of interest (advertising), and the scientific credibility of web-based diet- and nutrition-related information written in Japanese. Nutrition professionals and experts should take these findings seriously because exposure to nutritional information that lacks context or seems contradictory can lead to confusion and backlash among consumers. However, more research is needed to draw firm conclusions about the accuracy and quality of web-based diet- and nutrition-related content and whether similar results can be obtained in other major mass media or social media outlets and even other languages.