An appraisal of published usability evaluations of electronic health records via systematic review

A continuous usability evaluation of an electronic medication administration record application

RATIONALE, AIMS, AND OBJECTIVES

The complexity of an electronic medication administration record (eMAR) has been underestimated by most designers in the past. Usability issues, such as poorly designed user application flow in eMAR, are therefore of vital importance, since they can have a negative impact on nursing activities and result in poor outcomes. The purpose of this study was to evaluate the usability of an eMAR application during its development.

METHODS

A usability evaluation was conducted during the development of the eMAR application. Two usability methods were used: a heuristic evaluation complemented by usability testing. Each eMAR application version provided by the vendor was evaluated by 2 hospital pharmacists, who applied the heuristic method. They reviewed the eMAR tasks, detected usability problems and their heuristic violations, and rated the severity of the usability problems. Usability testing was used to assess the final application version by observing how 3 nurses interacted with the application.

RESULTS

Thirty-four versions were assessed before the eMAR application was considered usable. During the heuristic evaluation, the usability problems decreased from 46 unique usability problems in version 1 (V1) to 9 in version 34 (V34). In V1, usability problems were categorized into 154 heuristic violations, which decreased to 27 in V34. The average severity rating also decreased from major usability problem (2.96) to no problem (0.23). During usability testing, the 3 nurses did not encounter new usability problems.

CONCLUSION

A thorough heuristic evaluation is a good method for obtaining a usable eMAR application. This evaluation points key areas for improvement and decreases usability problems and their severity.

Toward Meaningful Care Plan Clinical Decision Support: Feasibility and Effects of a Simulated Pilot Study

BACKGROUND

Clinical decision support (CDS) tools-with easily understood and actionable information, at the point of care-are needed to help registered nurses (RNs) make evidence-based decisions. Not clear are the optimal formats of CDS tools. Thorough, preclinical testing is desirable to avoid costly errors associated with premature implementation in electronic health records.

OBJECTIVE

The aims of this study were to determine feasibility of the protocol designed to compare multiple CDS formats and evaluate effects of numeracy and graph literacy on RN adoption of best practices and care planning time in a simulated environment.

METHODS

In this pilot study, 60 RNs were randomly assigned to one of four CDS conditions (control, text, text + graph, and text + table) and asked to adjust the plan of care for two patient scenarios over three shifts. Fourteen best practices were identified for the two patients and sent as suggestions with evidence to the three CDS groups. Best practice adoption rates, care planning time, and their relationship to the RN's numeracy and graph literacy scores were assessed.

RESULTS

CDS groups had a higher adoption rate of best practices (p < .001) across all shifts and decreased care planning time in shifts 2 (p = .01) and 3 (p = .02) compared with the control group. Higher numeracy and graph literacy were associated with shorter care planning times under text + table (p = .05) and text + graph (p = .01) conditions. No significant differences were found between the three CDS groups on adoption rate and care planning time.

DISCUSSION

This pilot study shows the feasibility of our protocol. Findings show preliminary evidence that CDS improves the efficiency and effectiveness of care planning decisions and that the optimal format may depend on individual RN characteristics. We recommend a study with sufficient power to compare different CDS formats and assess the impact of potential covariates on adoption rates and care planning time.

Are We There Yet? Human Factors Knowledge and Health Information Technology - the Challenges of Implementation and Impact

Objective: To review the developments in human factors (HF) research on the challenges of health information technology (HIT) implementation and impact given the continuing incidence of usability problems and unintended consequences from HIT development and use. Methods: A search of PubMed/Medline and Web of Science® identified HF research published in 2015 and 2016. Electronic health records (EHRs) and patient-centred HIT emerged as significant foci of recent HF research. The authors selected prominent papers highlighting ongoing HF and usability challenges in these areas. This selective rather than systematic review of recent HF research highlights these key challenges and reflects on their implications on the future impact of HF research on HIT. Results: Research provides evidence of continued poor design, implementation, and usability of HIT, as well as technology-induced errors and unintended consequences. The paper highlights support for: (i) strengthening the evidence base on the benefits of HF approaches; (ii) improving knowledge translation in the implementation of HF approaches during HIT design, implementation, and evaluation; (iii) increasing transparency, governance, and enforcement of HF best practices at all stages of the HIT system development life cycle. Discussion and Conclusion: HF and usability approaches are yet to become embedded as integral components of HIT development, implementation, and impact assessment. As HIT becomes ever-more pervasive including with patients as end-users, there is a need to expand our conceptualisation of the problems to be addressed and the suite of tactics and strategies to be used to calibrate our pro-active involvement in its improvement.

Iterative User Interface Design for Automated Sequential Organ Failure Assessment Score Calculator in Sepsis Detection

Background

The new sepsis definition has increased the need for frequent sequential organ failure assessment (SOFA) score recalculation and the clerical burden of information retrieval makes this score ideal for automated calculation.

Objective

The aim of this study was to (1) estimate the clerical workload of manual SOFA score calculation through a time-motion analysis and (2) describe a user-centered design process for an electronic medical record (EMR) integrated, automated SOFA score calculator with subsequent usability evaluation study.

Methods

First, we performed a time-motion analysis by recording time-to-task-completion for the manual calculation of 35 baseline and 35 current SOFA scores by 14 internal medicine residents over a 2-month period. Next, we used an agile development process to create a user interface for a previously developed automated SOFA score calculator. The final user interface usability was evaluated by clinician end users with the Computer Systems Usability Questionnaire.

Results

The overall mean (standard deviation, SD) time-to-complete manual SOFA score calculation time was 61.6 s (33). Among the 24% (12/50) usability survey respondents, our user-centered user interface design process resulted in >75% favorability of survey items in the domains of system usability, information quality, and interface quality.

Conclusions

Early stakeholder engagement in our agile design process resulted in a user interface for an automated SOFA score calculator that reduced clinician workload and met clinicians’ needs at the point of care. Emerging interoperable platforms may facilitate dissemination of similarly useful clinical score calculators and decision support algorithms as “apps.” A user-centered design process and usability evaluation should be considered during creation of these tools.

A hospital-wide transition from paper to digital problem-oriented clinical notes. A descriptive history and cross-sectional survey of use, usability, and satisfaction

OBJECTIVES

To evaluate the use, usability, and physician satisfaction of a locally developed problem-oriented clinical notes application that replaced paper-based records in a large Dutch university medical center.

METHODS

Using a clinical notes database and an application event log file and a cross-sectional survey of usability, authors retrospectively analyzed system usage for medical specialties, users, and patients over 4 years. A standardized questionnaire measured usability. Authors analyzed the effects of sex, age, professional experience, training hours, and medical specialty on user satisfaction via univariate analysis of variance. Authors also examined the correlation between user satisfaction in relation to users' intensity of use of the application.

RESULTS

In total 1,793 physicians used the application to record progress notes for 219,755 patients. The overall satisfaction score was 3.2 on a scale from 1 (highly dissatisfied) to 5 (highly satisfied). A statistically significant difference occurred in satisfaction by medical specialty, but no statistically significant differences in satisfaction took place by sex, age, professional experience, or training hours. Intensity of system use did not correlate with physician satisfaction.

CONCLUSIONS

By two years after the start of the implementation, all medical specialties utilized the clinical notes application. User satisfaction was neutral (3.2 on a 1-5 scale). Authors believe that the significant factors facilitating this transition mirrored success factors reported by other groups: a generic, consistent, and transparent design of the application; intensive collaboration; continuous monitoring; and an incremental rollout.

User-centered design to improve clinical decision support in primary care

BACKGROUND

A growing literature has demonstrated the ability of user-centered design to make clinical decision support systems more effective and easier to use. However, studies of user-centered design have rarely examined more than a handful of sites at a time, and have frequently neglected the implementation climate and organizational resources that influence clinical decision support. The inclusion of such factors was identified by a systematic review as "the most important improvement that can be made in health IT evaluations."

OBJECTIVES

(1) Identify the prevalence of four user-centered design practices at United States Veterans Affairs (VA) primary care clinics and assess the perceived utility of clinical decision support at those clinics; (2) Evaluate the association between those user-centered design practices and the perceived utility of clinical decision support.

METHODS

We analyzed clinic-level survey data collected in 2006-2007 from 170 VA primary care clinics. We examined four user-centered design practices: 1) pilot testing, 2) provider satisfaction assessment, 3) formal usability assessment, and 4) analysis of impact on performance improvement. We used a regression model to evaluate the association between user-centered design practices and the perceived utility of clinical decision support, while accounting for other important factors at those clinics, including implementation climate, available resources, and structural characteristics. We also examined associations separately at community-based clinics and at hospital-based clinics.

RESULTS

User-centered design practices for clinical decision support varied across clinics: 74% conducted pilot testing, 62% conducted provider satisfaction assessment, 36% conducted a formal usability assessment, and 79% conducted an analysis of impact on performance improvement. Overall perceived utility of clinical decision support was high, with a mean rating of 4.17 (±.67) out of 5 on a composite measure. "Analysis of impact on performance improvement" was the only user-centered design practice significantly associated with perceived utility of clinical decision support, b=.47 (p<.001). This association was present in hospital-based clinics, b=.34 (p<.05), but was stronger at community-based clinics, b=.61 (p<.001).

CONCLUSIONS

Our findings are highly supportive of the practice of analyzing the impact of clinical decision support on performance metrics. This was the most common user-centered design practice in our study, and was the practice associated with higher perceived utility of clinical decision support. This practice may be particularly helpful at community-based clinics, which are typically less connected to VA medical center resources.