Usability Testing and Adaptation of the Pediatric Cardiovascular Risk Reduction Clinical Decision Support Tool

Methodologies for Evaluating the Usability of Rehabilitation Technologies Aimed at Supporting Shared Decision-Making: Scoping Review

BACKGROUND

The field of rehabilitation has seen a recent rise in technologies to support shared decision-making (SDM). Usability testing during the design process of SDM technologies is needed to optimize adoption and realize potential benefits. There is variability in how usability is defined and measured. Given the complexity of usability, a thorough examination of the methodologies used to measure usability to develop the SDM technologies used in rehabilitation care is needed.

OBJECTIVE

This scoping review aims to answer the following research questions: which methods and measures have been used to produce knowledge about the usability of rehabilitation technologies aimed at supporting SDM at the different phases of development and implementation? Which parameters of usability have been measured and reported?

METHODS

This review followed the Arksey and O'Malley framework. An electronic search was performed in the Ovid MEDLINE, Embase, CINAHL, and PsycINFO databases from January 2005 up to November 2020. In total, 2 independent reviewers screened all retrieved titles, abstracts, and full texts according to the inclusion criteria and extracted the data. The International Organization for Standardization framework was used to define the scope of usability (effectiveness, efficiency, and satisfaction). The characteristics of the studies were outlined in a descriptive summary. Findings were categorized based on usability parameters, technology interventions, and measures of usability.

RESULTS

A total of 38 articles were included. The most common SDM technologies were web-based aids (15/33, 46%). The usability of SDM technologies was assessed during development, preimplementation, or implementation, using 14 different methods. The most frequent methods were questionnaires (24/38, 63%) and semistructured interviews (16/38, 42%). Satisfaction (27/38, 71%) was the most common usability parameter mapped to types of SDM technologies and usability evaluation methods. User-centered design (9/15, 60%) was the most frequently used technology design framework.

CONCLUSIONS

The results from this scoping review highlight the importance and the complexity of usability evaluation. Although various methods and measures were shown to be used to evaluate the usability of technologies to support SDM in rehabilitation, very few evaluations used in the included studies were found to adequately span the selected usability domains. This review identified gaps in usability evaluation, as most studies (24/38, 63%) relied solely on questionnaires rather than multiple methods, and most questionnaires simply focused on the usability parameter of satisfaction. The consideration of end users (such as patients and clinicians) is of particular importance for the development of technologies to support SDM, as the process of SDM itself aims to improve patient-centered care and integrate both patient and clinician voices into their rehabilitation care.

Barriers and Facilitators to the Use of Clinical Decision Support Systems in Primary Care: A Mixed-Methods Systematic Review

PURPOSE

To identify and quantify the barriers and facilitators to the use of clinical decision support systems (CDSSs) by primary care professionals (PCPs).

METHODS

A mixed-methods systematic review was conducted using a sequential synthesis design. PubMed/MEDLINE, PsycInfo, Embase, CINAHL, and the Cochrane library were searched in July 2021. Studies that evaluated CDSSs providing recommendations to PCPs and intended for use during a consultation were included. We excluded CDSSs used only by patients, described as concepts or prototypes, used with simulated cases, and decision supports not considered as CDSSs. A framework synthesis was performed according to the HOT-fit framework (Human, Organizational, Technology, Net Benefits), then a quantitative synthesis evaluated the impact of the HOT-fit categories on CDSS use.

RESULTS

A total of 48 studies evaluating 45 CDSSs were included, and 186 main barriers or facilitators were identified. Qualitatively, barriers and facilitators were classified as human (eg, perceived usefulness), organizational (eg, disruption of usual workflow), and technological (eg, CDSS user-friendliness), with explanatory elements. The greatest barrier to using CDSSs was an increased workload. Quantitatively, the human and organizational factors had negative impacts on CDSS use, whereas the technological factor had a neutral impact and the net benefits dimension a positive impact.

CONCLUSIONS

Our findings emphasize the need for CDSS developers to better address human and organizational issues, in addition to technological challenges. We inferred core CDSS features covering these 3 factors, expected to improve their usability in primary care.

Combined Expert and User-Driven Usability Assessment of Trauma Decision Support Systems Improves User-Centered Design

BACKGROUND

Trauma clinical decision support systems improve adherence with evidence-based practice but suffer from poor usability and the lack of a user-centered design. The objective of this study was to compare the effectiveness of user and expert-driven usability testing methods to detect usability issues in a rib fracture clinical decision support system and identify guiding principles for trauma clinical decision support systems.

METHODS

A user-driven and expert-driven usability investigation was conducted using a clinical decision support system developed for patients with rib fractures. The user-driven usability evaluation was as follows: 10 clinicians were selected for simulation-based usability testing using snowball sampling, and each clinician completed 3 simulations using a video-conferencing platform. End-users participated in a novel team-based approach that simulated realistic clinical workflows. The expert-driven heuristic evaluation was as follows: 2 usability experts conducted a heuristic evaluation of the clinical decision support system using 10 common usability heuristics. Usability issues were identified, cataloged, and ranked for severity using a 4-level ordinal scale. Thematic analysis was utilized to categorize the identified usability issues.

RESULTS

Seventy-nine usability issues were identified; 63% were identified by experts and 48% by end-users. Notably, 58% of severe usability issues were identified by experts alone. Only 11% of issues were identified by both methods. Five themes were identified that could guide the design of clinical decision support systems-transparency, functionality and integration into workflow, automated and noninterruptive, flexibility, and layout and appearance. Themes were preferentially identified by different methods.

CONCLUSION

We found that a dual-method usability evaluation involving usability experts and end-users drastically improved detection of usability issues over single-method alone. We identified 5 themes to guide trauma clinical decision support system design. Performing usability testing via a remote video-conferencing platform facilitated multi-site involvement despite a global pandemic.

User-Centered Design to Enhance mHealth Systems for Individuals With Dexterity Impairments: Accessibility and Usability Study

Background

Mobile health systems have been shown to be useful in supporting self-management by promoting adherence to schedules and longitudinal health interventions, especially in people with disabilities. The Interactive Mobile Health and Rehabilitation (iMHere) system was developed to empower people with disabilities and those with chronic conditions with supports needed for self-management and independent living. Since the first iteration of the iMHere 1.0 app, several studies have evaluated the accessibility and usability of the system. Potential opportunities to improve and simplify the user interface were identified, and the iMHere modules were redesigned accordingly.

Objective

In this study, we aim to evaluate the usability of the redesigned modules within the iMHere 1.0 app.

Methods

We evaluated the original and redesigned iMHere modules—MyMeds and SkinCare. The Purdue Pegboard Test was administered to assess the participants’ dexterity levels. Participants were then asked to perform a set of tasks using both the original and redesigned MyMeds and SkinCare modules to assess their efficiency and effectiveness. Usability was measured using the Telehealth Usability Questionnaire to evaluate 10 new accessibility features that were added to the redesigned app. Participants were also asked which version they preferred.

Results

In total, 24 participants with disabilities and varying degrees of dexterity impairments completed the entire study protocol. Participants displayed improved efficiency and effectiveness when using the redesigned modules compared with the original modules. The participants also reported improved usability and preferred the redesigned modules.

Conclusions

This study demonstrated that the iMHere system became more efficient, effective, and usable for individuals with dexterity impairments after redesigning it according to user-centered principles.

User-centered and theory-based design of a professional training program on shared decision-making with older adults living with neurocognitive disorders: a mixed-methods study

BACKGROUND

We know little about the best approaches to design training for healthcare professionals. We thus studied how user-centered and theory-based design contribute to the development of a distance learning program for professionals, to increase their shared decision-making (SDM) with older adults living with neurocognitive disorders and their caregivers.

METHODS

In this mixed-methods study, healthcare professionals who worked in family medicine clinics and homecare services evaluated a training program in a user-centered approach with several iterative phases of quantitative and qualitative evaluation, each followed by modifications. The program comprised an e-learning activity and five evidence summaries. A subsample assessed the e-learning activity during semi-structured think-aloud sessions. A second subsample assessed the evidence summaries they received by email. All participants completed a theory-based questionnaire to assess their intention to adopt SDM. Descriptive statistical analyses and qualitative thematic analyses were integrated at each round to prioritize training improvements with regard to the determinants most likely to influence participants' intention.

RESULTS

Of 106 participants, 98 completed their evaluations of either the e-learning activity or evidence summary (93%). The professions most represented were physicians (60%) and nurses (15%). Professionals valued the e-learning component to gain knowledge on the theory and practice of SDM, and the evidence summaries to apply the knowledge gained through the e-learning activity to diverse clinical contexts. The iterative design process allowed addressing most weaknesses reported. Participants' intentions to adopt SDM and to use the summaries were high at baseline and remained positive as the rounds progressed. Attitude and social influence significantly influenced participants' intention to use the evidence summaries (P < 0.0001). Despite strong intention and the tailoring of tools to users, certain factors external to the training program can still influence the effective use of these tools and the adoption of SDM in practice.

CONCLUSIONS

A theory-based and user-centered design approach for continuing professional development interventions on SDM with older adults living with neurocognitive disorders and their caregivers appeared useful to identify the most important determinants of learners' intentions to use SDM in their practice, and validate our initial interpretations of learners' assessments during the subsequent evaluation round.

Provider Survey on Automated Clinical Decision Support System for Cardiovascular Risk Assessment

Despite progress made in establishing primary and secondary preventive strategies for cardiovascular diseases, there are significant gaps between guideline recommended strategies and implementation of recommendations in practice. A clinical decision support (CDS) system entitled CV Risk Profile was developed at Mayo Clinic Rochester as a targeted solution for this gap in preventive cardiovascular care. The system remained in use for 10 years until it became non-functional in 2018 during transition to a new electronic health record (EHR). This study investigated provider opinions regarding the cardiovascular disease CDS system while it was still in operation, to determine if there exists a provider reported need for a similar system to be developed for use within the new EHR.

Provider Survey on Automated Clinical Decision Support for Cardiovascular Risk Assessment

OBJECTIVE

To investigate provider opinions regarding a clinical decision support (CDS) system for cardiovascular risk assessment and for the creation of a replacement system.

METHODS

From March to April 2018, an invitation letter with a link to a self-administered web-based survey was sent via e-mail to 279 providers with primary appointment in the Department of Cardiovascular Medicine, Mayo Clinic, Rochester. The e-mail was sent to providers on March 8, 2018 and the survey closed on April 16, 2018.

RESULTS

One hundred providers responded to the survey yielding an overall response rate of 35.8%. Of these, 52 (52%) indicated they had used the cardiovascular (CV) risk profile CDS system and were classified as users and prompted to continue the survey. Among users, 42 (80.8%) indicated use of the CDS was either important (25; 48.1%) or very important (17; 32.7%) in their clinical practice; 45 (86.5%) responded that the system was very easy (17; 32.7%) or easy (28; 53.8%) to use. In addition, 48 (96.0%) users indicated that the CV risk profile supported their thought process at the point-of-care; 47 (97.9%) users indicated similar functionalities should be implemented into the new electronic health record system and 41 (85.4%) users reported new functionalities should also be incorporated.

CONCLUSIONS

For most users, the CDS system was easy to use and supported clinical thought process at the point-of-care. Users also felt their practice was supported and should continue to be supported by CDS systems providing individualized patient information at the point-of-care.

A Mobile Clinical Decision Support Tool for Pediatric Cardiovascular Risk-Reduction Clinical Practice Guidelines: Development and Description

Background

Widespread application of research findings to improve patient outcomes remains inadequate, and failure to routinely translate research findings into daily clinical practice is a major barrier for the implementation of any evidence-based guideline. Strategies to increase guideline uptake in primary care pediatric practices and to facilitate adherence to recommendations are required.

Objective

Our objective was to operationalize the US National Heart, Lung, and Blood Institute’s Integrated Guidelines for Cardiovascular Health and Risk Reduction in Children and Adolescents into a mobile clinical decision support (CDS) system for healthcare providers, and to describe the process development and outcomes.

Methods

To overcome the difficulty of translating clinical practice guidelines into a computable form that can be used by a CDS system, we used a multilayer framework to convert the evidence synthesis into executable knowledge. We used an iterative process of design, testing, and revision through each step in the translation of the guidelines for use in a CDS tool to support the development of 4 validated modules: an integrated risk assessment; a blood pressure calculator; a body mass index calculator; and a lipid management instrument.

Results

The iterative revision process identified several opportunities to improve the CDS tool. Operationalizing the integrated guideline identified numerous areas in which the guideline was vague or incorrect and required more explicit operationalization. Iterative revisions led to workable solutions to problems and understanding of the limitations of the tool.

Conclusions

The process and experiences described provide a model for other mobile CDS systems that translate written clinical practice guidelines into actionable, real-time clinical recommendations.