Bridging the Gap Between Academic Research and Pragmatic Needs in Usability: A Hybrid Approach to Usability Evaluation of Health Care Information Systems

Developing and Testing the Usability of a Novel Child Abuse Clinical Decision Support System: Mixed Methods Study

BACKGROUND

Despite the impact of physical abuse on children, it is often underdiagnosed, especially among children evaluated in emergency departments (EDs). Electronic clinical decision support (CDS) can improve the recognition of child physical abuse.

OBJECTIVE

We aimed to develop and test the usability of a natural language processing-based child abuse CDS system, known as the Child Abuse Clinical Decision Support (CA-CDS), to alert ED clinicians about high-risk injuries suggestive of abuse in infants' charts.

METHODS

Informed by available evidence, a multidisciplinary team, including an expert in user design, developed the CA-CDS prototype that provided evidence-based recommendations for the evaluation and management of suspected child abuse when triggered by documentation of a high-risk injury. Content was customized for medical versus nursing providers and initial versus subsequent exposure to the alert. To assess the usability of and refine the CA-CDS, we interviewed 24 clinicians from 4 EDs about their interactions with the prototype. Interview transcripts were coded and analyzed using conventional content analysis.

RESULTS

Overall, 5 main categories of themes emerged from the study. CA-CDS benefits included providing an extra layer of protection, providing evidence-based recommendations, and alerting the entire clinical ED team. The user-centered, workflow-compatible design included soft-stop alert configuration, editable and automatic documentation, and attention-grabbing formatting. Recommendations for improvement included consolidating content, clearer design elements, and adding a hyperlink with additional resources. Barriers to future implementation included alert fatigue, hesitancy to change, and concerns regarding documentation. Facilitators of future implementation included stakeholder buy-in, provider education, and sharing the test characteristics. On the basis of user feedback, iterative modifications were made to the prototype.

CONCLUSIONS

With its user-centered design and evidence-based content, the CA-CDS can aid providers in the real-time recognition and evaluation of infant physical abuse and has the potential to reduce the number of missed cases.

Practice facilitation for scale up of clinical decision support for hypertension management: study protocol for a cluster randomized control trial

BACKGROUND

Only half of patients with hypertension have adequately controlled blood pressure. Clinical decision support (CDS) has the potential to overcome barriers to delivering guideline-recommended care and improve hypertension management. However, optimal strategies for scaling CDS have not been well established, particularly in small, independent primary care practices which often lack the resources to effectively change practice routines. Practice facilitation is an implementation strategy that has been shown to support process changes. Our objective is to evaluate whether practice facilitation provided with hypertension-focused CDS can lead to improvements in blood pressure control for patients seen in small primary care practices.

METHODS/DESIGN

We will conduct a cluster randomized control trial to compare the effect of hypertension-focused CDS plus practice facilitation on BP control, as compared to CDS alone. The practice facilitation intervention will include an initial training in the CDS and a review of current guidelines along with follow-up for coaching and integration support. We will randomize 46 small primary care practices in New York City who use the same electronic health record vendor to intervention or control. All patients with hypertension seen at these practices will be included in the evaluation. We will also assess implementation of CDS in all practices and practice facilitation in the intervention group.

DISCUSSION

The results of this study will inform optimal implementation of CDS into small primary care practices, where much of care delivery occurs in the U.S. Additionally, our assessment of barriers and facilitators to implementation will support future scaling of the intervention.

CLINICALTRIALS

gov Identifier: NCT05588466.

Novel Note Templates to Enhance Signal and Reduce Noise in Medical Documentation: a Prospective Improvement Study

BACKGROUND

The introduction of electronic workflows has allowed for the flow of raw un-contextualized clinical data into medical documentation. As a result, many electronic notes have become replete of "noise" and deplete of clinically significant "signals". There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating.

OBJECTIVE

To describe the development and impact of a novel set of templates designed to change the flow of information in medical documentation.

METHODS

This is a multi-hospital nonrandomized prospective improvement study conducted on the Inpatient General Internal Medicine Service across three hospital campuses at the New York University (NYU) Langone Health System. A group of physician leaders representing each campus met biweekly for six months. The output of these meetings included 1) a conceptualization of the note bloat problem as a dysfunction in information flow 2) a set of guiding principles for organizational documentation improvement 3) the design and build of novel electronic templates that reduced the flow of extraneous information into provider notes by providing link outs to best practice data visualizations and 4) a documentation improvement curriculum for inpatient medicine providers. Prior to go-live, pragmatic usability testing was performed with the new progress note template, and the overall user experience measured using the System Usability Scale (SUS). Primary outcomes measures after go-live include template utilization rate and note length in characters.

RESULTS

In usability testing amongst 22 medicine providers, the new progress note template averaged a usability score of 90.6/100 on the System Usability Scale. 77% of providers strongly agreed that the new template was easy to use. 68% strongly agreed that they would like to use the template frequently. In the three months after template implementation, General Internal Medicine providers wrote 65% of all inpatient notes with the new templates. During this period of time the organization saw a 46%, 47%, and 32% reduction in note length for general medicine progress notes, consults, and H&Ps, respectively, when compared to a baseline measurement period prior to interventions.

CONCLUSIONS

A bundled intervention that included deployment of novel templates for inpatient general medicine providers significantly reduced average note length on the clinical service. Templates designed to reduce the flow of extraneous information into provider notes performed well during usability testing, and these templates were rapidly adopted across all hospital campuses. Further research is needed to assess the impact of novel templates on note quality, provider efficiency and patient outcomes.

CLINICALTRIAL

Health information communication technology evaluation frameworks for pharmacist prescribing: A systematic scoping review

BACKGROUND

Information communication technology (ICT) is instrumental in pharmacists' current practice and emerging roles. One such role is prescribing, which requires the use of clinical guidelines and documentation of decision-making, commonly via ICT. Development and refinement of ICT should be guided by evaluation frameworks that describe or measure features of ICT and its implementation. In the context of pharmacist prescribing, these evaluation frameworks should be specific to health stakeholders and the pharmacy setting.

OBJECTIVES

To identify ICT evaluation frameworks from health-related literature and review frameworks relevant to the development, implementation, and evaluation of pharmacist prescribing.

METHODS

A database search of CINAHL, Cochrane Library, EMBASE, Medline (Ovid), ProQuest, Scopus, Web of Science and grey literature was conducted, using combinations of keywords relating to 'ICT', 'utilization', 'usability', and 'evaluation framework'. Abstracts and titles were screened according to inclusion criteria. Identified evaluation frameworks were critiqued for relevance to pharmacy practice.

RESULTS

Twenty-two articles were identified, describing the development or application of 20 evaluation frameworks. None of the frameworks was developed specifically for pharmacy practice. The Technology Acceptance Model (TAM), describing use behavior, behavior intention, perceived usefulness, and perceived ease of use, was the most widely utilized framework. The Information System Success (ISS) and Human-Organization and Technology Fit (HOT-fit) are notable evaluation frameworks that address user and organizational influences in health ICT utility, and factors of both can address the limitation of TAM.

CONCLUSIONS

The findings call for development of an agile evaluation framework for the system under review; however, this can prove difficult due to the heterogenicity and complexity of the healthcare system, particularly contemporary pharmacy practice. While the TAM appears useful to evaluate user attitudes and intentions towards ICT, its relevance to ICT in contemporary community pharmacy practice requires exploration.

Using Intervention Mapping to Develop an mHealth Intervention to Support Men Who Have Sex With Men Engaging in Chemsex (Budd): Development and Usability Study

BACKGROUND

Chemsex refers to the intentional use of drugs before or during sex among men who have sex with men (MSM). Engaging in chemsex has been linked to significant negative impacts on physical, psychological, and social well-being. However, no evidence-based support tools have addressed either these harms or the care needs of MSM who engage in chemsex.

OBJECTIVE

The purpose of this paper was to describe the development of a mobile health intervention (named Budd) using the intervention mapping protocol (IMP). Budd aims to support and inform MSM who participate in chemsex, reduce the negative impacts associated with chemsex, and encourage more reasoned participation.

METHODS

The IMP consists of 6 steps to develop, implement, and evaluate evidence-based health interventions. A needs assessment was carried out between September 2, 2019, and March 31, 2020, by conducting a literature study and in-depth interviews. Change objectives were selected based on these findings, after which theory-based intervention methods were selected. The first version of the intervention was developed in December 2020 and pilot-tested between February 1, 2021, and April 30, 2021. Adjustments were made based on the findings from this study. A separate article will be dedicated to the effectiveness study, conducted between October 15, 2021, and February 24, 2022, and implementation of the intervention. The Budd app went live in April 2022.

RESULTS

Budd aims to address individual factors and support chemsex participants in applying harm reduction measures when taking drugs (drug information, drug combination tool, and notebook), preparing for participation in a chemsex session (articles on chemsex, preparation tool, and event-specific checklist), planning sufficient time after a chemsex session to recover (planning tool), seeking support for their chemsex participation (overview of existing local health care and peer support services, reflection, personal statistics, and user testimonials), taking HIV medication or pre-exposure prophylaxis in a timely manner during a chemsex session (preparation tool), and contacting emergency services in case of an emergency and giving first aid to others (emergency information and personal buddy).

CONCLUSIONS

The IMP proved to be a valuable tool in the planning and development of the Budd app. This study provides researchers and practitioners with valuable information that may help them to set up their own health interventions.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR1-10.2196/39678.

Internet-based cognitive behavior therapy for eating disorders - Development and feasibility evaluation

BACKGROUND

Eating disorders (ED) are severe psychiatric conditions, characterized by decreased quality of life and high mortality. However, only a minority of patients with ED seek care and very few receive treatment. Internet-delivered cognitive behavioral therapy (ICBT) has the potential to increase access to evidence-based treatments.

AIMS

The aims of the present study were to (1) develop and evaluate the usability of an Internet-delivered guided self-help treatment based on Enhanced Cognitive Behavioral Therapy (ICBT-E) for patients with full or subthreshold bulimia nervosa (BN) or binge eating disorder (BED) with a user centered design process, and (2) to evaluate its feasibility and preliminary outcome in a clinical environment.

METHOD

The study was undertaken in two stages. In Stage I, a user-centered design approach was applied with iterative phases of prototype development and evaluation. Participants were eight clinicians and 30 individuals with current or previous history of ED. In Stage II, 41 patients with full or subthreshold BN or BED were recruited to a single-group open trial to evaluate the feasibility and preliminary outcome of ICBT-E. Primary outcome variables were diagnostic status and self-rated ED symptoms.

RESULTS

The user-centered design process was instrumental in the development of the ICBT-E, by contributing to improvements of the program and to the content being adapted to the needs and preferences of end-users. The overall usability of the program was found to be good. ICBT-E targets key maintaining factors in ED by introducing healthy eating patterns and addressing over-evaluation of weight and shape. The results indicate that ICBT-E, delivered in a clinical setting, is a feasible and promising treatment for full or subthreshold BN or BED, with a high level of acceptability observed and treatment completion of 73.2 %. Participation in ICBT-E was associated with significant symptom reductions in core ED symptomology, functional impairment as well as depressive symptoms, and the results were maintained at the 3-month follow-up.

CONCLUSIONS

ICBT-E was developed with end-users' preferences in mind, in accordance with the identified recommendations, and the program was perceived as usable by end-users. The study demonstrated the potential of ICBT-E, which marks a step forward in the effort to make powerful, empirically supported psychological interventions targeting ED more widely available and accessible.

Developing organizational learning for scaling-up community-based primary health care in Ghana

INTRODUCTION

Achieving effective community-based primary health care requires evidence for guiding strategic decisions that must be made. However, research processes often limit data collection to particular organizational levels or disseminate results to specific audiences. Decision-making that emerges can fail to account for the contrasting perspectives and needs of managers at each organizational level. The Ghana Health Service (GHS) addressed this problem with a multilevel and sequential research and action approach that has provided two decades of implementation learning for guiding community-based primary health care development.

METHOD

The GHS implementation research initiatives progressed from (i) a participatory pilot investigation to (ii) an experimental trial of strategies that emerged to (iii) replication research for testing scale-up, culminating in (iv) evidence-based scale-up of a national community-based primary health care program. A reform process subsequently repeated this sequence in a manner that involved stakeholders at the community, sub-district, district, and regional levels of the system. The conduct, interpretation, and dissemination of results that emerged comprised a strategy for achieving systems learning by conducting investigations in phases in conjunction with bottom-up knowledge capture, lateral exchanges for fostering peer learning at each system level, and top-down processes for communicating results as policy. Continuous accumulation of qualitative data on stakeholder reactions to operations at each organizational level was conducted in conjunction with quantitative monitoring of field operations.

RESULTS

Implementation policies were enhanced by results associated with each phase. A quasi-experiment for testing the reform process showed that scale-up of community-based primary health care was accelerated, leading to improvements in childhood survival and reduced fertility.

CONCLUSION

Challenges to system learning were overcome despite severe resource constraints. The integration of knowledge generation with ongoing management processes institutionalized learning for achieving evidence-driven program action.

Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials

BACKGROUND

Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools.

OBJECTIVE

This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care.

METHODS

A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior.

RESULTS

To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images.

CONCLUSIONS

These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS.

TRIAL REGISTRATION

Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.

How to Enhance Digital Support for Cross-Organisational Health Care Teams? A User-Based Explorative Study

Health care service provision of individualised treatment to an ageing population prone to chronic conditions and multimorbidities is threatened. There is a need for digitally supported care, that is, (1) person-centred, (2) integrated, and (3) proactive. The research project 3P, Patients and Professionals in Productive Teams, aimed to validate and verify the prerequisites for health care systems run with patient-centred service models. This paper presents an explorative study of the digital support of a cross-organisational health care team in Norway, providing services to elderly frail people with multimorbidities in hospital discharge transition. Qualitative research methods were employed, with interviews and observations to map and evaluate the information flow and the digital support of collaborative work across organisations. The evaluation showed a lacking interoperability between the digital systems and a limited support for cross-organisational teamwork, causing raised manual efforts to maintain the information flow. Tools for coordination and planning across organisations were lacking. To enhance the situation, principles for a cloud-based health portal are proposed with a shared workspace, teamwork functionality for cross-organisational health care teams, and automatic back-end synchronisation of stored information. The main implications of this paper lie in the proposed principles which are transferable to a multitude of clinical contexts, where ad-hoc based access to shared medical information is of importance for decision-making and life-saving treatment.

Evaluating Online Consumer Medication Information Systems: Comparative Online Usability Study

BACKGROUND

Medication is the most common intervention in health care, and the number of online consumer information systems within the pharmaceutical sector is increasing. However, online consumer information systems can be a barrier for users, imposing information asymmetries between stakeholders.

OBJECTIVE

The objective of this study was to quantify and compare the usability of an online consumer medication information system (OCMIS) against a reference implementation based on an interoperable information model for patients, physicians, and pharmacists.

METHODS

Quantitative and qualitative data were acquired from patients, physicians, and pharmacists in this online usability study. We administered 3 use cases and a post hoc questionnaire per user. Quantitative usability data including effectiveness (task success), efficiency (task time), and user satisfaction (system usability scale [SUS]) was complemented by qualitative and demographic data. Users evaluated 6 existing systems and 1 reference implementation of an OCMIS.

RESULTS

A total of 137 patients, 81 physicians, and 68 pharmacists participated in this study. Task success varied from 84% to 92% in patients, 66% to 100% in physicians, and 50% to 91% in pharmacists. Task completion time decreased over the course of the study for all but 2 OCMIS within the patient group. Due to an assumed nonnormal distribution of SUS scores, within-group comparison was done using the Kruskal-Wallis test. Patients showed differences in SUS scores (P=.02) and task time (P=.03), while physicians did not have significant differences in SUS scores (P=.83) and task time (P=.72). For pharmacists, a significant difference in SUS scores (P<.001) and task time (P=.007) was detected.

CONCLUSIONS

The vendor-neutral reference implementation based on an interoperable information model was proven to be a promising approach that was not inferior to existing solutions for patients and physicians. For pharmacists, it exceeded user satisfaction scores compared to other OCMIS. This data-driven approach based on an interoperable information model enables the development of more user-tailored features to increase usability. This fosters data democratization and empowers stakeholders within the pharmaceutical sector.

Computerized Clinical Decision Support System for Emergency Department-Initiated Buprenorphine for Opioid Use Disorder: User-Centered Design

BACKGROUND

Emergency departments (EDs) frequently care for individuals with opioid use disorder (OUD). Buprenorphine (BUP) is an effective treatment option for patients with OUD that can safely be initiated in the ED. At present, BUP is rarely initiated as a part of routine ED care. Clinical decision support (CDS) could accelerate adoption of ED-initiated BUP into routine emergency care.

OBJECTIVE

This study aimed to design and formatively evaluate a user-centered decision support tool for ED initiation of BUP for patients with OUD.

METHODS

User-centered design with iterative prototype development was used. Initial observations and interviews identified workflows and information needs. The design team and key stakeholders reviewed prototype designs to ensure accuracy. A total of 5 prototypes were evaluated and iteratively refined based on input from 26 attending and resident physicians.

RESULTS

Early feedback identified concerns with the initial CDS design: an alert with several screens. The timing of the alert led to quick dismissal without using the tool. User feedback on subsequent iterations informed the development of a flexible tool to support clinicians with varied levels of experience with the intervention by providing both one-click options for direct activation of care pathways and user-activated support for critical decision points. The final design resolved challenging navigation issues through targeted placement, color, and design of the decision support modules and care pathways. In final testing, users expressed that the tool could be easily learned without training and was reasonable for use during routine emergency care.

CONCLUSIONS

A user-centered design process helped designers to better understand users' needs for a Web-based clinical decision tool to support ED initiation of BUP for OUD. The process identified varying needs across user experience and familiarity with the protocol, leading to a flexible design supporting both direct care pathways and user-initiated decision support.

Innovating From Within: A Process Model for User-Centered Digital Development in Academic Medical Centers

Background

Design thinking and human-centered design approaches have become increasingly common in health care literature, particularly in relation to health information technology (HIT), as a pathway toward the development of usable, diffusible tools and processes. There is a need in academic medical centers tasked with digital innovation for a comprehensive process model to guide development that incorporates current industry trends, including design thinking and lean and agile approaches to digital development.

Objective

This study aims to describe the foundations and phases of our model for user-centered HIT development.

Methods

Based on our experience, we established an integrated approach and rigorous process for HIT development that leverages design thinking and lean and agile strategies in a pragmatic way while preserving methodological integrity in support of academic research goals.

Results

A four-phased pragmatic process model was developed for user-centered digital development in HIT.

Conclusions

The model for user-centered HIT development that we developed is the culmination of diverse innovation projects and represents a multiphased, high-fidelity process for making more creative, flexible, efficient, and effective tools. This model is a critical step in building a rigorous approach to HIT design that incorporates a multidisciplinary, pragmatic perspective combined with academic research practices and state-of-the-art approaches to digital product development to meet the unique needs of health care.