The McMaster Optimal Aging Portal: Usability Evaluation of a Unique Evidence-Based Health Information Website

The Teach-ABI Professional Development Module for Educators About Pediatric Acquired Brain Injury: Mixed Method Usability Study

BACKGROUND

Acquired brain injury (ABI) is a leading cause of death and disability in children and can lead to lasting cognitive, physical, and psychosocial outcomes that affect school performance. Students with an ABI experience challenges returning to school due in part to lack of educator support and ABI awareness. A lack of knowledge and training contribute to educators feeling unprepared to support students with ABI. Teach-ABI, an online professional development module, was created to enhance educators' ABI knowledge and awareness to best support students. Using a case-based approach, Teach-ABI explains what an ABI is, identifies challenges for students with ABI in the classroom, discusses the importance of an individualized approach to supporting students with ABI, and describes how to support a student with an ABI in the classroom.

OBJECTIVE

This study aims to assess the usability of and satisfaction with Teach-ABI by elementary school educators. The following questions were explored: (1) Can elementary school teachers use and navigate Teach-ABI?, (2) Are the content and features of Teach-ABI satisfactory?, and (3) What modifications are needed to improve Teach-ABI?

METHODS

Elementary school educators currently employed or in training to be employed in Ontario elementary schools were recruited. Using Zoom, individual online meetings with a research team member were held, where educators actively reviewed Teach-ABI. Module usability was evaluated through qualitative analysis of think-aloud data and semistructured interviews, direct observation, user success rate during task completion, and the System Usability Scale (SUS) scores. The usability benchmark selected was 70% of participants performing more than half of module tasks independently.

RESULTS

A total of 8 female educators participated in the study. Educators were classroom (n=7) and preservice (n=1) teachers from public (n=7) and private (n=1) school boards. In terms of task performance, more than 85% of participants (ie, 7/8) independently completed 10 out of 11 tasks and 100% of participants independently completed 7 out of 11 tasks, demonstrating achievement of the module usability goal. The average overall SUS score was 86.25, suggesting a high satisfaction level with the perceived usability of Teach-ABI. Overall, participants found Teach-ABI content valuable, useful, and aligned with the realities of their profession. Participants appreciated the visual design, organization, and varying use of education strategies within Teach-ABI. Opportunities for enhancement included broadening content case examples of students with ABI and enhancing the accessibility of the content.

CONCLUSIONS

Validated usability measures combined with qualitative methodology revealed educators' high level of satisfaction with the design, content, and navigation of Teach-ABI. Educators engaged with the module as active participants in knowledge construction, as they reflected, questioned, and connected content to their experiences and knowledge. This study established strong usability and satisfaction with Teach-ABI and demonstrated the importance of usability testing in building online professional development modules.

Evaluation of a Digital Health Initiative in Illicit Substance Use: Cross-sectional Survey Study

Background

The Cracks in the Ice (CITI) community toolkit was developed to provide evidence-based, up-to-date information and resources about crystal methamphetamine to Australians. Given the high rates of internet use in the community and the potential for misinformation, CITI has the potential to play an important role in improving knowledge and challenging misconceptions surrounding crystal methamphetamine.

Objective

This study aims to determine (1) whether the CITI toolkit is achieving its aim of disseminating evidence-based information and resources to people who use crystal methamphetamine, their family and friends, health professionals, and the general community and (2) examine the association between the use of CITI and the knowledge and attitudes about crystal methamphetamine.

Methods

A cross-sectional web-based survey, open to Australian residents (aged ≥18 years), was conducted from November 2018 to March 2019. People who had previously visited the website (referred to as “website visitors” in this study) and those who had not (“naïve”) were recruited. At baseline, knowledge, attitudes, and demographics were assessed. CITI website visitors then completed a series of site evaluation questions, including the System Usability Scale (SUS), and naïve participants were asked to undertake a guided site tour of a replicated version of the site before completing the evaluation questions and repeating knowledge and attitude scales.

Results

Of a total 2108 participants, 564 (26.7%) reported lifetime use of crystal methamphetamine, 434 (20.6%) were family/friends, 288 (13.7%) were health professionals, and 822 (38.9%) were community members. The average SUS score was 73.49 (SD 13.30), indicating good site usability. Health professionals reported significantly higher SUS scores than community members (P=.02) and people who used crystal methamphetamine (P<.001). Website visitors had significantly higher baseline knowledge than naïve participants (P<.001). Among naïve participants, knowledge scores increased following exposure to the website (mean 15.2, SE 0.05) compared to baseline (mean 14.4, SE 0.05; P<.001). The largest shifts in knowledge were observed for items related to prevalence, legal issues, and the effects of the drug. Stigmatizing attitude scores among the naïve group were significantly lower following exposure to CITI (mean 41.97, SE 0.21) compared to baseline (mean 44.3, SE 0.21; P<.001).

Conclusions

This study provides an innovative evaluation of a national eHealth resource. CITI is achieving its aim of disseminating evidence-based, nonstigmatizing, and useful information and resources about crystal methamphetamine to key end user groups and has received good usability scores across its target groups. Interaction with CITI led to immediate improvements in knowledge about crystal methamphetamine and a decrease in stigmatizing attitudes. CITI demonstrates the important role of digital information and support platforms for translating evidence into practice and improving knowledge and reducing stigma.

Impact of a Knowledge Translation Intervention on Physical Activity and Mobility in Older Adults (the Move4Age Study): Randomized Controlled Trial

Background

The McMaster Optimal Aging Portal (the Portal) was launched in 2014 as a knowledge translation (KT) tool to increase access to evidence-based health information.

Objective

The purpose of this study was to understand if and how dissemination of mobility information through the Portal impacts physical activity (PA) in older adults.

Methods

In this randomized controlled trial, participants (n=510) were assigned to a 12-week mobility-focused KT intervention or self-serve control group. The intervention included weekly email alerts and a study-specific social media hashtag linking to mobility-focused Portal materials. The control group was able to access the Portal on their own but did not receive targeted KT strategies. Participants completed questionnaires (including the Rapid Assessment of Physical Activity to quantify PA) at baseline, end of the study, and 3-month follow-up.

Results

Participants were predominantly female (430/510, 84.3%), mean age 64.7 years, with no baseline differences between groups. Over half (277/510, 54.3%) of the participants were classified as “active” at baseline. There was no significant between-group difference in the PA category. Overall, both groups increased their PA with improvements maintained at 3-month follow-up (P<.001). In planned subgroup analyses, the KT intervention had a significant effect for those with poor or fair baseline self-rated health (P=.03).

Conclusions

No differences were found between those who received the targeted intervention and a control group with self-serve access to the Portal, except in subgroups with low self-rated health. Both groups did report increases in PA that were sustained beyond participation in a research study. Findings suggest that different KT strategies may be needed for different types of users, with more intense interventions being most impactful for certain groups (ie, those with lower self-rated health).

Trial Registration

ClinicalTrials.gov NCT02947230; https://clinicaltrials.gov/ct2/show/NCT02947230

Evaluation of an online knowledge translation intervention to promote cancer risk reduction behaviours: findings from a randomized controlled trial

Background

Many cancers are preventable through lifestyle modification; however, few adults engage in behaviors that are in line with cancer prevention guidelines. This may be partly due to the mixed messages on effective cancer prevention strategies in popular media. The goal of the McMaster Optimal Aging Portal (the Portal) is to increase access to trustworthy health information. The purpose of this study was to explore if and how knowledge translation strategies to disseminate cancer prevention evidence using the Portal influence participants’ knowledge, intentions and health behaviors related to cancer risk.

Methods

Adults ≥40 years old, with no cancer history were randomized to a 12-week intervention (weekly emails and social media posts) or control group. Quantitative data on knowledge, intentions and behaviors (physical activity, diet, alcohol consumption and use of tobacco products) were collected at baseline, end of study and 3 months later. Participant engagement was assessed using Google Analytics, and participant satisfaction through open-ended survey questions and semi-structured interviews.

Results

Participants (n = 557, mean age 64.9) were predominantly retired (72%) females (81%). Knowledge of cancer prevention guidelines was higher in the intervention group at end of study only (+ 0.3, p = 0.01). Intentions to follow cancer prevention guidelines increased in both groups, with no between-group differences. Intervention participants reported greater light-intensity physical activity at end of study (+ 0.7 vs. 0.1, p = 0.03), and reduced alcohol intake at follow u (− 0.2 vs. + 0.3, p < 0.05), but no other between-group differences were found. Overall satisfaction with the Portal and intervention materials was high.

Conclusions

Dissemination of evidence-based cancer prevention information through the Portal results in small increases in knowledge of risk-reduction strategies and with little to no impact on self-reported health behaviours, except in particular groups. Further tailoring of knowledge translation strategies may be needed to see more meaningful change in knowledge and health behaviours.

Trial registration

ClinicalTrials.gov NCT03186703, June 14, 2017.

Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study

Background

Patient experience surveys (PESs) often include informative free-text comments, but with no way of systematically, efficiently and usefully analysing and reporting these. The National Cancer Patient Experience Survey (CPES), used to model the approach reported here, generates > 70,000 free-text comments annually.

Main aim

To improve the use and usefulness of PES free-text comments in driving health service changes that improve the patient experience.

Secondary aims

(1) To structure CPES free-text comments using rule-based information retrieval (IR) (‘text engineering’), drawing on health-care domain-specific gazetteers of terms, with in-built transferability to other surveys and conditions; (2) to display the results usefully for health-care professionals, in a digital toolkit dashboard display that drills down to the original free text; (3) to explore the usefulness of interdisciplinary mixed stakeholder co-design and consensus-forming approaches in technology development, ensuring that outputs have meaning for all; and (4) to explore the usefulness of Normalisation Process Theory (NPT) in structuring outputs for implementation and sustainability.

Design

A scoping review, rapid review and surveys with stakeholders in health care (patients, carers, health-care providers, commissioners, policy-makers and charities) explored clinical dashboard design/patient experience themes. The findings informed the rules for the draft rule-based IR [developed using half of the 2013 Wales CPES (WCPES) data set] and prototype toolkit dashboards summarising PES data. These were refined following mixed stakeholder, concept-mapping workshops and interviews, which were structured to enable consensus-forming ‘co-design’ work. IR validation used the second half of the WCPES, with comparison against its manual analysis; transferability was tested using further health-care data sets. A discrete choice experiment (DCE) explored which toolkit features were preferred by health-care professionals, with a simple cost–benefit analysis. Structured walk-throughs with NHS managers in Wessex, London and Leeds explored usability and general implementation into practice.

Key outcomes

A taxonomy of ranked PES themes, a checklist of key features recommended for digital clinical toolkits, rule-based IR validation and transferability scores, usability, and goal-oriented, cost–benefit and marketability results. The secondary outputs were a survey, scoping and rapid review findings, and concordance and discordance between stakeholders and methods.

Results

(1) The surveys, rapid review and workshops showed that stakeholders differed in their understandings of the patient experience and priorities for change, but that they reached consensus on a shortlist of 19 themes; six were considered to be core; (2) the scoping review and one survey explored the clinical toolkit design, emphasising that such toolkits should be quick and easy to use, and embedded in workflows; the workshop discussions, the DCE and the walk-throughs confirmed this and foregrounded other features to form the toolkit design checklist; and (3) the rule-based IR, developed using noun and verb phrases and lookup gazetteers, was 86% accurate on the WCPES, but needs modification to improve this and to be accurate with other data sets. The DCE and the walk-through suggest that the toolkit would be well accepted, with a favourable cost–benefit ratio, if implemented into practice with appropriate infrastructure support.

Limitations

Small participant numbers and sampling bias across component studies. The scoping review studies mostly used top-down approaches and focused on professional dashboards. The rapid review of themes had limited scope, with no second reviewer. The IR needs further refinement, especially for transferability. New governance restrictions further limit immediate use.

Conclusions

Using a multidisciplinary, mixed stakeholder, use of co-design, proof of concept was shown for an automated display of patient experience free-text comments in a way that could drive health-care improvements in real time. The approach is easily modified for transferable application.

Future work

Further exploration is needed of implementation into practice, transferable uses and technology development co-design approaches.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

A Mobility-Focused Knowledge Translation Randomized Controlled Trial to Improve Physical Activity: Process Evaluation of the Move4Age Study

Background

Maintaining physical activity and physical function is important for healthy aging. We recently completed a randomized controlled trial of a targeted knowledge translation (KT) intervention delivered through the McMaster Optimal Aging Portal with the goal to increase physical activity and physical mobility in middle-aged and older adults, with results reported elsewhere.

Objective

The purpose of this process evaluation study is to explore which KT strategies were used by both intervention and control group participants, as well as the intervention groups’ engagement, satisfaction, and perceived usefulness of the targeted KT intervention.

Methods

Data on engagement with the intervention materials were gathered quantitatively through Google Analytics and Hootsuite throughout the intervention. Qualitative data were collected through a combination of open-ended surveys and qualitative interviews with a subset of participants at the end of the study to further understand engagement, satisfaction, and usefulness of the KT strategies.

Results

Throughout the intervention period, engagement with content delivered through weekly emails was highest, and participants rated email content most favorably in both surveys and interviews. Participants were generally satisfied with the intervention, noting the ease of participating and the distillation of information in an easy-to-access format being beneficial features. Participants who did not find the intervention useful were those with already high levels of baseline physical activity or physical function and those who were looking for more specific or individualized content.

Conclusions

This process evaluation provides insight into our randomized controlled trial findings and provides information that can be used to improve future online KT interventions.

Trial Registration

ClinicalTrials.gov NCT02947230; https://clinicaltrials.gov/ct2/show/nct02947230 (Archived by WebCite at http://www.webcitation.org/78t4tR8tM)

A Tool That Assesses the Evidence, Transparency, and Usability of Online Health Information: Development and Reliability Assessment

Background

The internet is commonly used by older adults to obtain health information and this trend has markedly increased in the past decade. However, studies illustrate that much of the available online health information is not informed by good quality evidence, developed in a transparent way, or easy to use. Furthermore, studies highlight that the general public lacks the skills necessary to distinguish between online products that are credible and trustworthy and those that are not. A number of tools have been developed to assess the evidence, transparency, and usability of online health information; however, many have not been assessed for reliability or ease of use.

Objective

The first objective of this study was to determine if a tool assessing the evidence, transparency, and usability of online health information exists that is easy and quick to use and has good reliability. No such tool was identified, so the second objective was to develop such a tool and assess it for reliability when used to assess online health information on topics of relevant to optimal aging.

Methods

An electronic database search was conducted between 2002 and 2012 to identify published papers describing tools that assessed the evidence, transparency, and usability of online health information. Papers were retained if the tool described was assessed for reliability, assessed the quality of evidence used to create online health information, and was quick and easy to use. When no one tool met expectations, a new instrument was developed and tested for reliability. Reliability between two raters was assessed using the intraclass correlation coefficient (ICC) for each item at two time points. SPSS Statistics 22 software was used for statistical analyses and a one-way random effects model was used to report the results. The overall ICC was assessed for the instrument as a whole in July 2015. The threshold for retaining items was ICC>0.60 (ie, “good” reliability).

Results

All tools identified that evaluated online health information were either too complex, took a long time to complete, had poor reliability, or had not undergone reliability assessment. A new instrument was developed and assessed for reliability in April 2014. Three items had an ICC<0.60 (ie, “good” reliability). One of these items was removed (“minimal scrolling”) and two were retained but reworded for clarity. Four new items were added that assessed the level of research evidence that informed the online health information and the tool was retested in July 2015. The total ICC score showed excellent agreement with both single measures (ICC=0.988; CI 0.982–0.992) and average measures (ICC=0.994; CI 0.991–0.996).

Conclusions

The results of this study suggest that this new tool is reliable for assessing the evidence, transparency, and usability of online health information that is relevant to optimal aging.

If You Build It, Who Will Come? A Description of User Characteristics and Experiences With the McMaster Optimal Aging Portal

Objectives: The McMaster Optimal Aging Portal (the Portal) aims to increase access to evidence-based health information. We would now like to understand who uses the Portal, why, and for what, and elicit feedback and suggestions for future initiatives. Methods: An online survey of users collected data on demographics, eHealth literacy, Internet use, information-seeking behavior, site acceptability and perceived impact on health behaviors, participant satisfaction, and suggestions for improvements using mixed methods. Results: Participants (n = 163, age 69.8 ± 8.6 years) were predominantly female (76%), married (67%), retired (80%), and well-educated with very good/excellent health (55%). The Portal was easy to use (83%) and relevant (80%), with 68% intending to, and 48% having changed behavior after using the Portal. A number of suggestions for improvement were obtained. Discussion: A better understanding of users' characteristics, needs, and preferences will allow us to improve content, target groups who are not engaging with the Portal, and plan future directions.

McMaster Optimal Aging Portal: an evidence-based database for geriatrics-focused health professionals

Objective

The objective of this work was to provide easy access to reliable health information based on good quality research that will help health care professionals to learn what works best for seniors to stay as healthy as possible, manage health conditions and build supportive health systems. This will help meet the demands of our aging population that clinicians provide high quality care for older adults, that public health professionals deliver disease prevention and health promotion strategies across the life span, and that policymakers address the economic and social need to create a robust health system and a healthy society for all ages.

Results

The McMaster Optimal Aging Portal’s (Portal) professional bibliographic database contains high quality scientific evidence about optimal aging specifically targeted to clinicians, public health professionals and policymakers. The database content comes from three information services: McMaster Premium LiteratUre Service (MacPLUS™), Health Evidence™ and Health Systems Evidence. The Portal is continually updated, freely accessible online, easily searchable, and provides email-based alerts when new records are added. The database is being continually assessed for value, usability and use. A number of improvements are planned, including French language translation of content, increased linkages between related records within the Portal database, and inclusion of additional types of content. While this article focuses on the professional database, the Portal also houses resources for patients, caregivers and the general public, which may also be of interest to geriatric practitioners and researchers.

Patient web portals, disease management, and primary prevention

BACKGROUND

Efforts aimed at health care reform and continued advances in information technologies have prompted interest among providers and researchers in patient web portals. Patient web portals are password-protected online websites that offer the patients 24-hour access to personal health information from anywhere with an Internet connection.

METHODS

This article, which is based upon bibliographic searches in PubMed, reviews important developments in web portals for primary and secondary disease prevention, including patient web portals tethered to electronic medical records, disease-specific portals, health disparities, and health-related community web portals.

RESULTS

Although findings have not been uniformly positive, several studies of the effectiveness of health care system patient portals in chronic disease management have shown promising results with regard to patient outcomes. Patient web portals have also shown promising results in increasing adherence with screening recommendations. Racial and ethnic minorities, younger persons, and patients who are less educated or have lower health literacy have been found to be less likely to use patient portals.

CONCLUSION

Additional studies are needed of the utility and effectiveness of different elements of web portals for different patient populations. This should include additional diseases and health topics such as smoking cessation and weight management.

Rare Diseases on the Internet: An Assessment of the Quality of Online Information

BACKGROUND

The importance of the Internet as a medium for publishing and sharing health and medical information has increased considerably during the last decade. Nonetheless, comprehensive knowledge and information are scarce and difficult to find, especially for rare diseases. Additionally, the quality of health or medical information about rare diseases is frequently difficult to assess for the patients and their family members.

OBJECTIVE

The aim of this study is to assess the quality of information on the Internet about rare diseases. Additionally, the study aims to evaluate if the quality of information on rare diseases varies between different information supplier categories.

METHODS

A total of 13 quality criteria for websites providing medical information about rare diseases were transferred to a self-disclosure questionnaire. Identified providers of information on the Internet about rare diseases were invited to fill out the questionnaire. The questionnaire contained questions about the information provider in general (eg, supplier category, information category, language, use of quality certificates, and target group) and about quality aspects that reflect the 13 quality criteria. Differences in subgroup analyses were performed using t tests.

RESULTS

We identified 693 websites containing information about rare diseases. A total of 123 questionnaires (17.7%) were completely filled out by the information suppliers. For the remaining identified suppliers (570/693, 82.3%), the questionnaires were filled out by the authors based on the information available on their website. In many cases, the quality of websites was proportionally low. Furthermore, subgroup analysis showed no statistically significant differences between the quality of information provided by support group/patient organization compared to medical institution (P=.19). The quality of information by individuals (patient/relative) was significantly lower compared to information provided by support group/patient organization (P=.001), medical institution (P=.009), and other associations and sponsoring bodies (P=.001) as well.

CONCLUSIONS

Overall, the quality of information on the Internet about rare diseases is low. Quality certificates are rarely used and important quality criteria are often not fulfilled completely. Additionally, some information categories are underrepresented (eg, information about psychosocial counseling, social-legal advice, and family planning). Nevertheless, due to the high amount of information provided by support groups, this study shows that these are extremely valuable sources of information for patients suffering from a rare disease and their relatives.

Adopting Quality Criteria for Websites Providing Medical Information About Rare Diseases

Background

The European Union considers diseases to be rare when they affect less than 5 in 10,000 people. It is estimated that there are between 5000 and 8000 different rare diseases. Consistent with this diversity, the quality of information available on the Web varies considerably. Thus, quality criteria for websites about rare diseases are needed.

Objective

The objective of this study was to generate a catalog of quality criteria suitable for rare diseases.

Methods

First, relevant certificates and quality recommendations for health information websites were identified through a comprehensive Web search. Second, all considered quality criteria of each certification program and catalog were examined, extracted into an overview table, and analyzed by thematic content. Finally, an interdisciplinary expert group verified the relevant quality criteria.

Results

We identified 9 quality certificates and criteria catalogs for health information websites with 304 single criteria items. Through this, we aggregated 163 various quality criteria, each assigned to one of the following categories: thematic, technical, service, content, and legal. Finally, a consensus about 13 quality criteria for websites offering medical information on rare diseases was determined. Of these categories, 4 (data protection concept, imprint, creation and updating date, and possibility to contact the website provider) were identified as being the most important for publishing medical information about rare diseases.

Conclusions

The large number of different quality criteria appearing within a relatively small number of criteria catalogs shows that the opinion of what is important in the quality of health information differs. In addition, to define useful quality criteria for websites about rare diseases, which are an essential source of information for many patients, a trade-off is necessary between the high standard of quality criteria for health information websites in general and the limited provision of information about some rare diseases. Finally, transparently presented quality assessments can help people to find reliable information and to assess its quality.

User Experiences of the McMaster Optimal Aging Portal's Evidence Summaries and Blog Posts: Usability Study

BACKGROUND

Evidence summaries and blogs can support evidence-informed healthy aging, by presenting high-quality health research evidence in plain language for a nonprofessional (citizen) audience.

OBJECTIVE

Our objective was to explore citizens' perceptions about the usability of evidence summaries and blog posts on the Web-based McMaster Optimal Aging Portal.

METHODS

Twenty-two citizens (aged 50 years and older) and informal caregivers participated in a qualitative study using a think-aloud method and semistructured interviews. Eleven interviews were conducted in person, 7 over the telephone, and 4 by Skype.

RESULTS

We identified themes that fell under 4 user-experience categories: (1) desirability: personal relevance, (2) understandability: language comprehension, grasping the message, dealing with uncertainty, (3) usability: volume of information, use of numbers, and (4) usefulness: intention to use, facility for sharing.

CONCLUSIONS

Participants recognized that high-quality evidence on aging was valuable. Their intended use of the information was influenced by how much it applied to their own health circumstances or those of a loved one. Some specific formatting features that were preferred included consistent layout, content organized by subheadings, catchy titles, numerical information summarized in a table, and inclusion of a glossary.