Systematic assessment of information about surgical urinary stone treatment on YouTube

Accuracy and Readability of Kidney Stone Patient Information Materials Generated by a Large Language Model Compared to Official Urologic Organizations

OBJECTIVE

To compare the readability and accuracy of large language model generated patient information materials (PIMs) to those supplied by the American Urological Association (AUA), Canadian Urological Association (CUA), and European Association of Urology (EAU) for kidney stones.

METHODS

PIMs from AUA, CUA, and EAU related to nephrolithiasis were obtained and categorized. The most frequent patient questions related to kidney stones were identified from an internet query and input into GPT-3.5 and GPT-4. PIMs and ChatGPT outputs were assessed for accuracy and readability using previously published indexes. We also assessed changes in ChatGPT outputs when a reading level was specified (grade 6).

RESULTS

Readability scores were better for PIMs from the CUA (grade level 10-12), AUA (8-10), or EAU (9-11) compared to the chatbot. GPT-3.5 had the worst readability scores at grade 13-14 and GPT-4 was likewise less readable than urologic organization PIMs with scores of 11-13. While organizational PIMs were deemed to be accurate, the chatbot had high accuracy with minor details omitted. GPT-4 was more accurate in general stone information, dietary and medical management of kidney stones topics in comparison to GPT-3.5, while both models had the same accuracy in the surgical management of nephrolithiasis topics.

CONCLUSION

Current PIMs from major urologic organizations for kidney stones remain more readable than publicly available GPT outputs, but they are still higher than the reading ability of the general population. Of the available PIMs for kidney stones, those from the AUA are the most readable. Although Chatbot outputs for common kidney stone patient queries have a high degree of accuracy with minor omitted details, it is important for clinicians to understand their strengths and limitations.

Assessing the educational value of laparoscopic radical nephrectomy videos on YouTube®: A comparative analysis of short versus long videos

INTRODUCTION

To evaluate the quality of laparoscopic radical nephrectomy videos and determine the extent to which they are informative and educational for healthcare professionals.

PATIENTS AND METHODS

We used the YouTube® search engine to search for the term 'laparoscopic radical nephrectomy' with time filters of 4-20 min (Group 1) and >20 min (Group 2) and then sorted the results uploaded chronologically before January 2023. One hundred videos were analysed for each group. The reliability of the videos was assessed using the Journal of American Medical Association (JAMA) Benchmark Criteria and DISCERN questionnaire scores (DISCERN). Educational quality was assessed using the Global Quality Score (GQS) and a 20-item objective scoring system (OSS) for laparoscopic nephrectomy. The popularity of the videos was evaluated using the video power index (VPI).

RESULTS

The mean video duration was 8.9 ± 4.3 min in Group 1 and 52.02 ± 31.09 min in Group 2 (P < 0.001). The mean JAMA (2.49 ± 0.61) and OSS scores (60 ± 12.3) were higher in Group 2 than in Group 1, while no significant difference was observed in the mean GQS (2.53 ± 0.7, 2.39 ± 0.88, respectively) between the groups (P < 0.001, P = 0.039, P = 0.131, respectively).

CONCLUSION

While the standardisation of surgical videos published on YouTube® and the establishment of auditing mechanisms do not seem plausible, high total OSS, periprocedural OSS, and VPI scores, and high OSS, JAMAS, GQS and DISCERN scores in long videos indicate that such videos offer a greater contribution to education.

YouTube as a Source of Patient and Trainee Education in Vascular Surgery: A Systematic Review

Objective

Due to its video based approach, YouTube has become a widely accessed educational resource for patients and trainees. This systematic review characterised and evaluated the peer reviewed literature investigating YouTube as a source of patient or trainee education in vascular surgery.

Data sources

A comprehensive literature search was conducted using EMBASE, MEDLINE, and Ovid HealthStar from inception until 19 January 2023. All primary studies and conference abstracts evaluating YouTube as a source of vascular surgery education were included.

Review methods

Video educational quality was analysed across several factors, including pathology, video audience, and length.

Results

Overall, 24 studies were identified examining 3 221 videos with 123.1 hours of content and 37.1 million views. Studies primarily examined YouTube videos on diabetic foot care (7/24, 29%), peripheral arterial disease (3/24, 13%), carotid artery stenosis (3/24, 13%), varicose veins (3/24, 13%), and abdominal aortic aneurysm (2/24, 8%). Video educational quality was analysed using standardised assessment tools, author generated scoring systems, or global author reported assessment of quality. Six studies assessed videos for trainee education, while 18 studies evaluated videos for patient education. Among the 20 studies which reported on the overall quality of educational content, 10/20 studies deemed it poor, and 10/20 studies considered it fair, with 53% of studies noting poor educational quality for videos intended for patients and 40% of studies noting poor educational quality in videos intended for trainees. Poor quality videos had more views than fair quality videos (mean 27 348, 95% CI 15 154-39 543 views vs. 11 372, 95% CI 3 115-19 629 views, p = .030).

Conclusion

The overall educational quality of YouTube videos for vascular surgery patient and trainee education is suboptimal. There is significant heterogeneity in the quality assessment tools used in their evaluation. A standardised approach to online education with a consistent quality assessment tool is required to better support online patient and trainee education in vascular surgery.

Assessment of YouTube Videos about Urethral Stricture and Its Management

PURPOSE

To evaluate the reliability, transparency, usability and technical quality of videos about urethral stricture and its management in the most common video-sharing website "YouTube".

METHODS

We performed a video search on YouTube using the keywords "Urethral Stricture", "Internal Urethrotomy" and "Urethroplasty". After excluding the videos that did not meet the inclusion criteria, demographic data of each video was recorded and assessed with the validated DISCERN questionnaire, the Journal of the American Medical Association Benchmark Score (JAMAs) and the Global Quality Score (GQS). Additionally, the Urethral Stricture Video Score was developed by two experienced surgeons to assess the technical aspects. Videos were also compared with a view to their source of upload.

RESULTS

A total of 177 videos were analysed. Videos from "University" and "Other" had a significantly higher view ratio and video power index (VPI) than the rest (all p<0.05). For all mentioned scores, even though the video sources of "Society" and "University" had significantly higher scores than the rest (all p<0.05), all scores remained at a low to moderate level at each group.

CONCLUSION

Even though universities, societies and organisations tend to upload better videos about the management of urethral stricture in terms of general quality, reliability, transparency and technical aspects, the online visual content about urethral stricture and its management needs to be improved in order to provide more reliable and qualified information for both clinicians and patients.

Evaluating YouTube as a Source of Education for Patients Undergoing Surgery: A Systematic Review

OBJECTIVE

The objective of this systematic review is to characterize the peer-reviewed literature investigating YouTube as a source of patient education for patients undergoing surgery.

SUMMARY BACKGROUND DATA

YouTube is the largest online video sharing platform and has become a substantial source of health information that patients are likely to access before surgery, yet there has been no systematic assessment of peer-reviewed studies. A comprehensive literature search was conducted using EMBASE, MEDLINE, and Ovid HealthStar from inception through to December of 2021.

METHODS

All primary studies evaluating YouTube as a source of patient education relating to surgical procedures (general, cardiac, urology, otolaryngology, plastic, vascular) were included. Study screening and data extraction occurred in duplicate with two reviewers. Characteristics extracted included video length, view count, upload source, overall video educational quality, and quality of individual studies.

RESULTS

Among 6,453 citations, 56 studies were identified that examined 6,797 videos with 547 hours of content and 1.39 billion views. There were 49 studies that evaluated the educational quality of the videos. A total of 43 quality assessment tools were used, with each study using a mean of 1.90 assessment tools. Per the global rating for assessments, 34/49 studies (69%) concluded that the overall quality of educational content was poor.

CONCLUSIONS

While the impact of non-peer-reviewed YouTube videos on patient knowledge for surgery is unclear, the large amount of online content suggests that they are in demand. The overall educational content of these videos is poor, however, and there is substantial heterogeneity in the quality assessment tools used in their evaluation. A peer-reviewed and standardized approach to online education with video content is needed to better support patients.

Conflict of interest and funding in health communication on social media: a systematic review

OBJECTIVES

To synthesise the available evidence on the reporting of conflicts of interest (COI) by individuals posting health messages on social media, and on the reporting of funding sources of studies cited in health messages on social media.

DATA SOURCES

MEDLINE (OVID) (2005-March 2022), Embase (2005-March 2022) and Google Scholar (2005-August 2022), supplemented with a review of reference lists and forward citation tracking.

DESIGN

Reviewers selected eligible studies and abstracted data in duplicate and independently. We appraised the quality of the included studies using the Mixed Methods Appraisal Tool. We summarised the results in both narrative and tabular formats. We followed the PRISMA 2020 checklist for reporting our study.

RESULTS

Of a total of 16 645 retrieved citations, we included 17 eligible studies. The frequency of reporting of conflicts of interest varied between 0% and 60%, but it was mostly low. In addition, a significant proportion, ranging between 15% and 80%, of healthcare professionals using social media have financial relationships with industry. However, three studies assessed the proportion of conflicts of interest of physicians identified through Open Payment Database but not reported by the authors. It was found that 98.7-100% of these relationships with industry are not reported when communicating health-related information. Also, two studies showed that there is evidence of a potential association between COI and the content of posting. No data was found on the reporting of funding sources of studies cited in health messages on social media.

CONCLUSIONS

While a significant proportion of healthcare professionals using social media have financial relationships with industry, lack of reporting on COI and undisclosed COI are common. We did not find studies on the reporting of funding sources of studies cited in health messages on social media.

TRIAL REGISTRATION

dx.doi.org/10.17504/protocols.io.5jyl8jj4rg2w/v1.

Assessing the quality of publicly available videos on MDMA-assisted psychotherapy for PTSD

BACKGROUND AND OBJECTIVES

Patients increasingly rely on the Internet for healthcare information. This study aimed to evaluate the quality of videos on 3,4-methylenedioxymethamphetamine (MDMA)-assisted psychotherapy for posttraumatic stress disorder (PTSD) on YouTube™.

METHODS

YouTube™ was searched for the terms "MDMA" and "PTSD." The 100 most viewed videos were analyzed using three standard quality measures: Global Quality Scores (GQS), JAMA benchmark, and DISCERN. Viewer engagement features and source of upload, video duration, inclusion of patient narrative and/or MD/DO/PhD, the mention of lack of Food and Drug Administration (FDA) approval, side effects, potential for abuse, and use in conjunction with psychotherapy were recorded.

RESULTS

The videos were of poor quality (mean GQS: 2.26 ± 0.94/5, JAMA: 1.96 ± 0.45/4, and DISCERN: 29.5 ± 8.2/80). A significant positive association was found between video quality and duration (GQS: r = .5857, p < .0001, JAMA: r = .279, p = .0409, DISCERN: r = .5783, p < .0001). Videos including an MD/DO/PhD had the highest scores (GQS: 2.87/5 [1.22], p = .006, DISCERN: 38.35/80 [13.32], p < .0003). A minority of videos were uploaded by academic institutions (1%); most were from professional organizations (29%). No correlation was found between quality and viewer engagement features-number of views, subscribers, likes/dislikes, or comments. A majority mentioned that MDMA must be used in conjunction with psychotherapy (85%) and is not FDA-approved (82%) for PTSD. Only 32% of videos mentioned risks or potential for abuse.

CONCLUSIONS

These findings highlight the need for better quality of online health material and an opportunity for involvement of healthcare professionals in the dissemination of accurate health information via content creation.

SCIENTIFIC SIGNIFICANCE

This is the first study to examine publicly available information on the use of MDMA for PTSD.

"YouTube™ as a source of information on placenta accreta: A quality analysis"

OBJECTIVE

To evaluate the quality of YouTube™ videos on placenta accreta and to investigate if they can be used as a reliable source of information.

METHODS

We queried YouTube™ with terms related to placenta accreta. Patient Education Materials Assessment Tool for audio-visual content (PEMAT A/V), Global Quality Score (GQS), DISCERN score and Misinformation tool were used to assess videos' quality content.

RESULTS

Sixty-four videos were suitable for the analyses. Of those, 42 (65.6%) and 22 (34.4%) were produces by healthcare works and others, respectively. The median PEMAT A/V Understandability and Actionability score was 75 and 66.7%, respectively. According to GQS, 31.2 and 45.3% videos were excellent or good and generally poor or poor, respectively. The median DISCERN score of section 1 was 27 (out of 40), of section 2 was 16 (out of 35) and of section 3 was 2 (out of 5). According to Misinformation tool, the worst described aspect was the one regarding the possible risk factor associated to placenta accreta. Performing the quality assessment according to video authoring entity, videos produced by healthcare workers harbored a higher quality content, relative to the others.

CONCLUSIONS

Currently, the overall consideration of YouTube™ video content on placenta accreta is low. Societies should invest new sources in producing higher quality videos to provide a helpful tool for physician during the counselling with patients.

Analysis of quality information provided by "Dr. YouTubeTM" on Phimosis

The objective of the current study was to evaluate the quality of the information provided in YouTube^TM videos on phimosis. The term “phimosis” was searched on YouTube^TM, and the Patient Education Materials Assessment Tool (PEMAT) for Audio/Visual Materials (Understandability and Actionability sections, good-quality score of minimum 70%) and misinformation scale (rated from 1 to 5) were used to assess video quality. Quality assessment was investigated over time. Of all, 60 were eligible for analysis. Healthcare providers were the authors of 75.0% of the videos, and 73.3% of the videos were patient-targeted. The median Understandability score was 42.9% (interquartile range [IQR]:34.5–58.9) and ranged from 28.6 to 42.9% (2013–2020). The median Actionability score was 50.0% (IQR:25.0–56.2) and ranged from 25.0 to 50.0% (2013–2020). The median misinformation score was 2.8/5 (IQR:1.6–3.6), and although the score fluctuated over time, the median score was 2.6 both in 2013 and in 2020. According to our results, although an increase of PEMAT over time was observed, the overall quality of the information uploaded on YouTube^TM is low. Therefore, at present, YouTube^TM cannot be recommended as a reliable source of information on phimosis. Video producers should upload higher-quality videos to help physicians and patients in the decision-making process.

Evaluation of YouTube Videos as a Source of Patient Information for Ureteric Stent Placement: A Quality Assessment Study

Objective

To assess the quality of YouTube videos on ureteric stent placement (USP) as a source of patient available.

Methods

YouTube was searched using search terms “DJ stenting,” “Double J stenting,” and “ureteric stenting.” The initial 100 videos displayed with each of the above mentioned search terms were scrutinized. The selected videos reviewed by 3 independent consultant urologists against a pre-agreed scoring system based upon European Association of Urology (EAU) patient information sheet on ureteric stent placement. The videos were scored qualitatively and quantitatively based on the scores achieved in various domains of the scoring Performa. Data was also collected for the number of views, likes, dislikes, and time duration of each video.

Results

A total of 22 videos which fulfilled the inclusion criteria were reviewed. All the videos were uploaded by healthcare organizations or healthcare websites. None of the videos were classified as “Good” based on reviewer scores and only one video was classified as “acceptable.” Fourteen videos were classified as “very poor” with a score of <5/20. General information about stents was described by majority of the studies whilst preoperative information, procedure description, danger signs, and follow up were scarcely described by most videos.

Conclusion

Majority of YouTube videos on USP are of poor overall quality and lack pertinent information. This calls for creation of comprehensive and unbiased videos for patient information on USP.

YouTube videos in Japanese as a source of information on nocturnal enuresis: A content-quality and reliability analysis

BACKGROUND

This study aimed to evaluate the quality and reliability of Japanese YouTube videos pertaining to nocturnal enuresis (NE).

METHODS

In this cross-sectional study, we performed a YouTube search using the keyword "Ya-nyou-shou" ('nocturnal enuresis' in Japanese). We considered the first 200 videos listed based on the YouTube default option. Videos that were irrelevant, concerned personal experiences or adult NE, had non-Japanese content, were advertisements, were duplicated, and those without audio were excluded. Video features and upload source were recorded. The Modified DISCERN, Journal of the American Medical Association (JAMA), and Global Quality Scale (GQS) scoring systems were used for analysis. Two independent pediatricians, specialists in nephrology and urology, completed the scoring. Correlation analysis was performed between video features and the three quality analysis scores.

RESULTS

In total, 72 videos were analyzed. The most common upload sources were physicians (40.3%) and non-physician health personnel (40.3%). The median modified DISCERN, JAMA, and GQS scores for the videos were 1 (lowest: 0; highest: 4), 2 (lowest: 1; highest: 3), and 2 (lowest: 1; highest: 4), respectively. The highest scores for video power index (VPI) were for independent users (6.43 points) and physicians (3.05 points). There were positive correlations between the VPI and video length and modified DISCERN and GQS scores. No video presenter disclosed conflicts of interest.

CONCLUSIONS

Most Japanese YouTube videos about NE were low quality. Healthcare experts should be encouraged to upload better quality content.

YouTubeTM as a source of information on bladder pain syndrome: A contemporary analysis

AIM

To evaluate the quality of YouTube™ videos on bladder pain syndrome (BPS) and to investigate whether they can be used as a reliable source of information.

METHODS

The search term "bladder pain syndrome" was used on YouTube^TM platform. The first 100 videos were selected. Patient Education Materials Assessment Tool for audio-visual content (PEMAT A/V), Global Quality Score (GQS), Misinformation tool, and DISCERN score were used to assess videos' quality content. Pearson's test was used to assess potential correlations between variables.

RESULTS

Seventy-nine videos were suitable for the analyses. The median PEMAT A/V Understandability score and PEMAT A/V Actionability score were 66.7% (interquartile range [IQR]: 46.2-100.0) and 75.0% (IQR: 37.5-100.0), respectively. According to GQS, 26 (32.9%), 32 (40.5%), 3 (3.8%), 15 (19.0%), and 3 (3.8%) videos were excellent, good, moderate, generally poor, and poor, respectively. According to Misinformation tool, of all videos, 81% (n = 64), 6.3% (n = 5), 5.1% (n = 4), 5.1% (n = 4), 2.5% (n = 2) had respectively no, very little, moderate, high, and extreme misinformation. The overall median DISCERN score ranged from 5.0 (IQR: 2.0-5.0) to 5.0 (IQR: 5.0-5.0). A positive statistically significant correlation was found between video length and PEMAT A/V Understandability (r = 0.27, p = 0.01), video length and PEMAT A/V Actionability (r = .26, p = 0.02), and video length and DISCERN Question 16 (r = 0.28, p = 0.01).

CONCLUSIONS

Nowaday, the overall quality of YouTube^TM videos on BPS have been evaluated good according to PEMAT A/V, GQS, Misinformation tool, and DISCERN score. It is possible to assume that YouTube^TM may be considered as a reliable source of information on BPS.

Radiotherapy or Surgery? Comparative, Qualitative Assessment of Online Patient Education Materials on Prostate Cancer

As multiple different treatment options are available for prostate cancer (PCa) and YouTube is commonly used as a source for medical information, we performed a systematic and comparative assessment of available videos guiding patients on their choice for the optimal treatment. An independent search for surgical therapy or radiotherapy of PCa on YouTube was performed and the 40 most viewed videos of both groups were analyzed. The validated DISCERN questionnaire and PEMAT were utilized to evaluate their quality and misinformation. The median overall quality of the videos was found to be low for surgery videos, while radiotherapy videos results reached a moderate quality. The median PEMAT understandability score was 60% (range 0-100%) for radiotherapy and 75% (range 40-100) for surgery videos. The radiotherapy videos contained less misinformation and were judged to be of higher quality. Summarized, the majority of the provided videos offer insufficient quality of content and are potentially subject to commercial bias without reports on possible conflict of interest. Thus, most of available videos on YouTube informing PCa patients about possible treatment methods are not suited for a balanced patient education or as a basis for the patient's decision.