Content and Quality of Websites for Patients With Chronic Kidney Disease: An Environmental Scan

Can large language models provide accurate and quality information to parents regarding chronic kidney diseases?

RATIONALE

Artificial Intelligence (AI) large language models (LLM) are tools capable of generating human-like text responses to user queries across topics. The use of these language models in various medical contexts is currently being studied. However, the performance and content quality of these language models have not been evaluated in specific medical fields.

AIMS AND OBJECTIVES

This study aimed to compare the performance of AI LLMs ChatGPT, Gemini and Copilot in providing information to parents about chronic kidney diseases (CKD) and compare the information accuracy and quality with that of a reference source.

METHODS

In this study, 40 frequently asked questions about CKD were identified. The accuracy and quality of the answers were evaluated with reference to the Kidney Disease: Improving Global Outcomes guidelines. The accuracy of the responses generated by LLMs was assessed using F1, precision and recall scores. The quality of the responses was evaluated using a five-point global quality score (GQS).

RESULTS

ChatGPT and Gemini achieved high F1 scores of 0.89 and 1, respectively, in the diagnosis and lifestyle categories, demonstrating significant success in generating accurate responses. Furthermore, ChatGPT and Gemini were successful in generating accurate responses with high precision values in the diagnosis and lifestyle categories. In terms of recall values, all LLMs exhibited strong performance in the diagnosis, treatment and lifestyle categories. Average GQ scores for the responses generated were 3.46 ± 0.55, 1.93 ± 0.63 and 2.02 ± 0.69 for Gemini, ChatGPT 3.5 and Copilot, respectively. In all categories, Gemini performed better than ChatGPT and Copilot.

CONCLUSION

Although LLMs provide parents with high-accuracy information about CKD, their use is limited compared with that of a reference source. The limitations in the performance of LLMs can lead to misinformation and potential misinterpretations. Therefore, patients and parents should exercise caution when using these models.

An evaluation of the quality of COVID-19 websites in terms of HON principles and using DISCERN tool

BACKGROUND

As many people relied on information from the Internet for official scientific or academically affiliated information during the COVID-19 pandemic, the quality of information on those websites should be good.

OBJECTIVE

The main purpose of this study was to evaluate a selection of COVID-19-related websites for the quality of health information provided.

METHOD

Using Google and Yahoo, 36 English language websites were selected, in accordance with the inclusion criteria. The two tools were selected for evaluation were the Health on the Net (HON) Code and the 16-item DISCERN tool.

RESULTS

Most websites (39%) were related to information for the public, and a small number of them (3%) concerned screening websites in which people could be informed of their possible condition by entering their symptoms. The result of the evaluation by the HON tool showed that most websites were reliable (53%), and 44% of them were very reliable. Based on the assessment results of the Likert-based 16-item DISCERN tool, the maximum and minimum values for the average scores of each website were calculated as 2.44 and 4.25, respectively.

CONCLUSION

Evaluation using two widely accepted tools shows that most websites related to COVID-19 are reliable and useful for physicians, researchers and the public.

Quality and accuracy of online nutrition-related information: a systematic review of content analysis studies

OBJECTIVE

This systematic review aimed to summarise the level of quality and accuracy of nutrition-related information on websites and social media and determine if quality and accuracy varied between websites and social media or publishers of information.

DESIGN

This systematic review was registered with PROSPERO (CRD42021224277). CINAHL, MEDLINE, Embase, Global Health and Academic Search Complete were systematically searched on 15 January 2021 to identify content analysis studies, published in English after 1989, that evaluated the quality and/or accuracy of nutrition-related information published on websites or social media. A coding framework was used to classify studies' findings about information quality and/or accuracy as poor, good, moderate or varied. The Academy of Nutrition and Dietetics Quality Criteria Checklist was used to assess the risk of bias.

SETTING

N/A.

PARTICIPANTS

N/A.

RESULTS

From 10 482 articles retrieved, sixty-four were included. Most studies evaluated information from websites (n 53, 82·8 %). Similar numbers of studies assessed quality (n 41, 64·1 %) and accuracy (n 47, 73·4 %). Almost half of the studies reported that quality (n 20, 48·8 %) or accuracy (n 23, 48·9 %) was low. Quality and accuracy of information were similar on social media and websites, however, varied between information publishers. High risk of bias in sample selection and quality or accuracy evaluations was a common limitation.

CONCLUSION

Online nutrition-related information is often inaccurate and of low quality. Consumers seeking information online are at risk of being misinformed. More action is needed to improve the public's eHealth and media literacy and the reliability of online nutrition-related information.

Didactical characteristics of Dutch websites about kidney transplantation targeted for kidney patients and living donors: An exploratory study

Objective

The aim was to explore the origin, content topics, teaching modes (instruction, interaction, and assessment), and corresponding social-epistemological dimensions (choices in knowledge transfer vs. knowledge building, and individual vs. group learning) of web-based information on kidney transplantation targeted for patients and living donors.

Methods

Dutch websites on kidney transplantation were retrieved using the search engine Google.nl. From 24 websites, 250 webpages were examined on origin, content topics, teaching modes, and corresponding social-epistemological dimensions.

Results

The majority of the websites had a professional organization as origin (20/24). The number and distribution of content topics varied among the websites. Of the 16 different teaching modes found, 11 were instructional, 4 were interactional, and 1 assessment mode was found. The websites offered almost exclusively teaching modes on individual and passive learning, whereas group learning and interactive knowledge building was hardly encountered.

Conclusion

The diversity in teaching modes and social-epistemological dimensions of Dutch websites on kidney transplantation targeted for patients and living donors is limited. The websites only provided a partial view on kidney transplantation; information regarding contact with others, e.g. support, was limited. A more balanced availability of teaching approaches and content topics is desirable to fit with the didactical goals to make well-considered health decisions.

Quality assessment of consumer-facing websites on sodium reduction

The purpose of this paper is to evaluate the quality of information and guidance on dietary sodium reduction available on consumer‐facing websites. Google Trends was used to identify the five most‐used search terms related to dietary sodium reduction. For each term, websites on the first two pages were collected (n = 18–20). Of 93 websites collected, 24 were excluded due to defective links, duplicate websites, or not being consumer‐focused. The remaining 69 websites were evaluated using a novel instrument, JHU‐SALT, that includes 14 questions on topics related to salt reduction. The questions are grouped into three domains (“information,” “guidance,” and “accuracy”). For each question, websites were scored using a 3‐step ordinal scale (“topic not addressed,” “topic somewhat addressed,” or “topic addressed adequately”). Only three of 14 JHU‐SALT questions were addressed adequately by a majority of websites. Many websites provided information on the adverse health effects of a high sodium diet (74%, n = 51) or mentioned intake recommendations (64%, n = 44). Information on fundamental concepts was largely missing. The majority of websites (80%, n = 55) provided information on lifestyle strategies to reduce blood pressure, but most did not provide guidance to help implement those strategies. While missing information was common, misinformation was uncommon. The DISCERN questionnaire was utilized as well. Consumers seeking information and guidance on dietary sodium reduction will find that most available websites provide accurate but limited information, and insufficient guidance on how to lower sodium intake. Websites that provide both relevant information and guidance are needed to help consumers effectively reduce dietary sodium.

An eHealth self-management intervention for adults with chronic kidney disease, My Kidneys My Health: a mixed-methods study

BACKGROUND

There is limited research of electronic tools for self-management for patients with chronic kidney disease (CKD). We sought to evaluate participant engagement, perceived self-efficacy and website usage in a preliminary evaluation of My Kidneys My Health, a patient-facing eHealth tool in Canada.

METHODS

We conducted an explanatory sequential mixed-methods study of adults with CKD who were not on kidney replacement therapy and who had access to My Kidneys My Health for 8 weeks. Outcomes included acceptance (measured by the Technology Acceptance Model), self-efficacy (measured by the Chronic Disease Self-Efficacy Scale [CDSES]) and website usage patterns (captured using Google Analytics). We analyzed participant interviews using qualitative content analysis.

RESULTS

Twenty-nine participants with CKD completed baseline questionnaires, of whom 22 completed end-of-study questionnaires; data saturation was achieved with 15 telephone interviews. Acceptance was high, with more than 70% of participants agreeing or strongly agreeing that the website was easy to use and useful. Of the 22 who completed end-of-study questionnaires, 18 (82%) indicated they would recommend its use to others and 16 (73%) stated they would use the website in the future. Average scores for website satisfaction and look and feel were 7.7 (standard deviation [SD] 2.0) and 8.2 (SD 2.0) out of 10, respectively. The CDSES indicated that participants gained an increase in CKD information. Interviewed participants reported that the website offered valuable information and interactive tools for patients with early or newly diagnosed CKD, or for those experiencing changes in health status. Popular website pages and interactive features included Food and Diet, What is CKD, My Question List and the Depression Screener.

INTERPRETATION

Participants indicated that the My Kidneys My Health website provided accessible content and tools that may improve self-efficacy and support in CKD self-management. Further evaluation of the website's effectiveness in supporting self-management among a larger, more heterogenous population is warranted.

Identifying individual enablers and barriers to the use of digital technology for the self-management of long-term conditions by older adults

Evidence suggests that much of the digital technology available and provided to older adults to enable self-management of long-term conditions is under-utilised. This research focuses on three conditions prevalent amongst older adults: diabetes, dementia and chronic kidney disease and explores the individual enablers and barriers to the use of digital self-management technology. The paper reports findings from a series of three systematic reviews of qualitative research (qualitative evidence syntheses). These reviews informed the design of a Delphi study. The first round of the Delphi involving 15 expert interviews is reported. The findings highlight common themes across the three conditions: how technology is used; barriers to use; assessing individual needs when selecting technology; support requirements; multi-functional self-management technologies; trust, privacy and data sharing; achieving accessible and aspirational design. Some emerging recommendations have been suggested to guide the design, and provision of technology to older adults. These will extended and refined through subsequent rounds of the Delphi method.

An evaluation of the quality of online perinatal depression information

BACKGROUND

During the perinatal period (including pregnancy and up to 12 months after childbirth), expectant and new mothers are at an elevated risk of developing depression. Inadequate knowledge about perinatal depression and treatment options may contribute to the low help-seeking rates exhibited by perinatal people. The Internet can be an accessible source of information about perinatal depression; however, the quality of this information remains to be evaluated. The purpose of this study was to assess the quality of perinatal depression information websites.

METHODS

After review, 37 websites were included in our sample. To assess overall website quality, we rated websites based on their reading level (Simple Measure of Gobbledegook; SMOG), information quality (DISCERN), usability (Patient Education Materials Assessment Tool; PEMAT), and visual design (Visual Aesthetics of Website Inventory; VisAWI).

RESULTS

Websites often exceeded the National Institute of Health's recommended reading level of grades 6-8, with scores ranging from 6.8 to 13.5. Website information quality ratings ranged from 1.8 to 4.3 out of 5, with websites often containing insufficient information about treatment choices. Website usability ratings were negatively impacted by the lack of information summaries, visual aids, and tangible tools. Visual design ratings ranged from 3.2 to 6.6 out of 7, with a need for more creative design elements to enhance user engagement.

CONCLUSIONS

This study outlines the characteristics of high-quality perinatal depression information websites. Our findings illustrate that perinatal depression websites are not meeting the needs of users in terms of reading level, information quality, usability, and visual design. Our results may be helpful in guiding healthcare providers to reliable, evidence-based online resources for their perinatal patients.

A Research Protocol for Implementation and Evaluation of a Patient-Focused eHealth Intervention for Chronic Kidney Disease

Self-management in chronic kidney disease (CKD) can slow disease progression; however, there are few tools available to support patients with early CKD. My Kidneys My Health is a patient-focused electronic health (eHealth) self-management tool developed by patients and caregivers. This study will investigate the implementation of My Kidneys My Health across primary care and general nephrology clinics. The study aims to: (1) identify and address barriers and facilitators that may impact implementation and sustainability of the website into routine clinical care; (2) evaluate implementation quality to inform spread and scale-up. We will conduct a multi-stage approach using qualitative methods, guided by the Quality Implementation Framework and using a qualitative content analysis approach. First, we will identify perceived barriers and facilitators to implementation and considerations for sustainability through interviews with clinicians, based on the Readiness Thinking Tool and the Long Term Success Tool. Analysis will be guided by the Consolidated Framework for Implementation Research and the Theoretical Domains Framework. Appropriate implementation strategies will be identified using the Expert Recommendations for Implementing Change compilation, and implementation plans will be developed based on Proctor's recommendations and the Action, Actor, Context, Target, Time framework. Finally, we will explore implementation quality guided by the RE-AIM framework. There is limited literature describing systematic approaches to implementing and sustaining patient-focused self-management tools into clinical care, in addition to employing tailored implementation strategies to promote adoption and sustainability. We aim to generate insights on how My Kidneys My Health can be integrated into clinical care and how to sustain use of patient-centric eHealth tools in clinical settings on a larger scale.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43477-022-00038-3.

A Proposed Approach for Conducting Studies That Use Data From Social Media Platforms

The prominence of social media in contemporary society has extended significantly into the health care arena, where both patients and health care providers have used social media platforms to gather, communicate, learn, and share medical content and personal experience in real time. The medical literature has also seen an exponential increase in the number of studies that use data derived from social media coverage of various medical issues and topics. In this guide, we present a step-by-step framework for health care professionals and researchers to conduct studies that use data from social media platforms. We present 6 overarching steps: focus on framing a question that is appropriate for social media evaluation, identification of social media outlet and selection criteria of content, systematic data extraction, assessment of quality of content and sources of bias, analysis of data, and interpretation of study findings. Each step is illustrated with published examples.

A Web-Based Self-Management Support Prototype for Adults With Chronic Kidney Disease (My Kidneys My Health): Co-Design and Usability Testing

Background

Supporting patients to self-manage their chronic kidney disease (CKD) has been identified as a research priority by patients with CKD and those who care for them. Self-management has been shown to slow CKD progression and improve the quality of life of individuals living with the disease. Previous work has identified a need for a person-centered, theory-informed, web-based tool for CKD self-management that can be individualized to a patient’s unique situation, priorities, and preferences. We addressed this gap using an integrated knowledge translation method and patient engagement principles.

Objective

The aim of this study is to conduct systematic co-design and usability testing of a web-based self-management prototype for adults with CKD (nondialysis and nontransplant) and their caregivers to enhance self-management support.

Methods

A multistep, iterative system development cycle was used to co-design and test the My Kidneys My Health prototype. The 3-step process included creating website features and content using 2 sequential focus groups with patients with CKD and caregivers, heuristic testing using the 10 heuristic principles by Nielsen, and usability testing through in-person 60-minute interviews with patients with CKD and their caregivers. Patients with CKD, caregivers, clinicians, researchers, software developers, graphic designers, and policy makers were involved in all steps of this study.

Results

In step 1, 18 participants (14 patients and 4 caregivers) attended one of the 2 sequential focus groups. The participants provided specific suggestions for simplifying navigation as well as suggestions to incorporate video, text, audio, interactive components, and visuals to convey information. A total of 5 reviewers completed the heuristic analysis (step 2), identifying items mainly related to navigation and functionality. Furthermore, 5 participants completed usability testing (step 3) and provided feedback on video production, navigation, features and functionality, and branding. Participants reported visiting the website repeatedly for the following features: personalized food tool, my health care provider question list, symptom guidance based on CKD severity, and medication advice. Usability was high, with a mean system usability score of 90 out of 100.

Conclusions

The My Kidneys My Health prototype is a systematically developed, multifaceted, web-based CKD self-management support tool guided by the theory and preferences of patients with CKD and their caregivers. The website is user friendly and provides features that improve user experience by tailoring the content and resources to their needs. A feasibility study will provide insights into the acceptability of and engagement with the prototype and identify preliminary patient-reported outcomes (eg, self-efficacy) as well as potential factors related to implementation. This work is relevant given the shift to virtual care during the current pandemic times and provides patients with support when in-person care is restricted.

Features of patient-education websites for patients with chronic kidney disease: an analysis of recommended websites

BACKGROUND

Chronic kidney disease (CKD) requires lifelong self-management. With the rise in access to the Internet, many CKD patients and their caregivers increasingly use the internet for information on CKD self-management. A recent environmental scan by Smekal et al. identified 11 CKD-related websites that covered the greatest number of content areas. This paper aims to evaluate these 11 selected websites in order to identify those that most effectively address content areas relevant to patients with CKD.

METHODS

Each website was assessed for information to 6 content areas: diet, physical activity, financial information, emotional support, general CKD information, and medication adherence. A three-tiered scoring metric was used in which a 0 was given if a content area was completely unaddressed, a (+) was given for a category that was generally addressed, and a (++) was given for a category that was addressed with actionable guidance.

RESULTS

While CKD information and diet were very comprehensively covered with scores of 11 (++) and 8 (++), respectively; physical activity, emotional support and medication adherence received the fewest (++) scores (3 for physical activity and five for both emotional support and medication adherence). For each content area, recommendations are made for websites that are particularly useful. Common themes for these highlighted websites include specific instructions, multiple modalities of information, downloadable and printable resources, and contact references for personal inquiries.

CONCLUSION

The recommended websites can help CKD patients and caregivers utilize the most applicable information for their specific self-management needs. Website improvements related to physical activity, emotional support, and financial information for persons with CKD are warranted.

Quality and Readability Assessment of Websites on Human Papillomavirus and Oropharyngeal Cancer

OBJECTIVES/HYPOTHESIS

The incidence of human papillomavirus-positive (HPV+) oropharyngeal cancer is rising, but public knowledge about this diagnosis remains low. This study aimed to investigate the quality and readability of online information about HPV+ oropharyngeal cancer.

STUDY DESIGN

Cross-sectional website analysis.

METHODS

This study conducted a total of 12 web searches across Google, Yahoo, and Bing to identify websites related to HPV+ oropharyngeal cancer. The QUality Evaluation Scoring Tool (QUEST) was used to measure quality based on seven website criteria. The Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL) were used to measure readability, with scores estimating the education level a reader would require to understand a piece of text. Readability improves as FRES increases and FKGL decreases.

RESULTS

Twenty-seven unique web pages were evaluated. The mean USA reading grade level as measured by FKGL was 10.42 (standard deviation = 1.54). There was an inverse relationship between quality and readability, with a significant positive correlation between QUEST score and FKGL (r = 0.343, P = .040) and a significant negative correlation between QUEST score and FRES (r = -0.537, P = .002).

CONCLUSIONS

With a mean USA reading grade level more than four grades above the American Medical Association's recommendation and results indicating that readability suffers as quality improves, these findings suggest that the currently available online information about HPV+ oropharyngeal cancer is insufficient. Improved patient education practices and resources about this diagnosis are needed.

LEVEL OF EVIDENCE

NA Laryngoscope, 131:87-94, 2021.