An appraisal of published usability evaluations of electronic health records via systematic review

Using Electronic Health Record Data for Occupational Therapy Health Services Research: Invited Commentary

Health services research (HSR) is a field of study that examines how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affect access to health care, the quality and cost of health care, and health and well-being. HSR approaches can help build the occupational therapy evidence base, particularly in relation to population health. Data from electronic health record (EHR) systems provide a rich resource for applying HSR approaches to examine the value of occupational therapy services. Transparency about data preparation procedures is important for interpreting results. Based on our findings, we describe a six-step cleaning protocol for preparing EHR and billing data from an inpatient rehabilitation facility for research and provide recommendations for the field based on our experience. Using and reporting similar strategies across studies will improve efficiency and transparency, and facilitate comparability of results.

Design and evaluation of a web-based electronic health record for amblyopia

Introduction

Amblyopia, or lazy eye, is a type of visual impairment in which the eyesight is not complete, even with the use of glasses. For the treatment of this disease, accurate and continuous examinations are needed. Nowadays, patient-centered care, by relying on web-based electronic records for amblyopia, has the potential to reduce treatment costs, increase the quality of care, and improve the safety and effectiveness of treatment. Therefore, the purpose of this study is to design and evaluate an Electronic Health Record (EHR) for patients with amblyopia.

Methods

The present study is applied developmental research. Using a Morgan table as a sampling tool, a straightforward random sampling technique selected 150 records from 1,500 records that were free of flaws. The design of the electronic version proceeded in a cascading manner so that after the design of each part, it was presented to the amblyopia experts, and if approved, the next part was designed. To design this EHR, the C# programming language and MySQL database were used. A system evaluation was performed by entering and recording patient information. For this purpose, the standard Questionnaire of User Interaction Satisfaction (QUIS), consisting of 18 questions, was used.

Results

According to the amblyopia EHR data elements, the data of physician and patient, examinations, website members, and members' roles were determined. After defining the fields and classes that explain the tables, the EHR was designed. The usability evaluation of the system showed that the mean selection of very good and good options by the users of EHRs was over 90%, indicating the patients' acceptance of web-based EHRs.

Conclusion

The design of an EHR for amblyopia is an effective step toward integrating and improving the information management of these patients. It will also enable the storage and retrieval of patients' information to reduce and facilitate the control of amblyopia complications.

Methods used to evaluate usability of mobile clinical decision support systems for healthcare emergencies: a systematic review and qualitative synthesis

Objective

The aim of this study was to determine the methods and metrics used to evaluate the usability of mobile application Clinical Decision Support Systems (CDSSs) used in healthcare emergencies. Secondary aims were to describe the characteristics and usability of evaluated CDSSs.

Materials and Methods

A systematic literature review was conducted using Pubmed/Medline, Embase, Scopus, and IEEE Xplore databases. Quantitative data were descriptively analyzed, and qualitative data were described and synthesized using inductive thematic analysis.

Results

Twenty-three studies were included in the analysis. The usability metrics most frequently evaluated were efficiency and usefulness, followed by user errors, satisfaction, learnability, effectiveness, and memorability. Methods used to assess usability included questionnaires in 20 (87%) studies, user trials in 17 (74%), interviews in 6 (26%), and heuristic evaluations in 3 (13%). Most CDSS inputs consisted of manual input (18, 78%) rather than automatic input (2, 9%). Most CDSS outputs comprised a recommendation (18, 78%), with a minority advising a specific treatment (6, 26%), or a score, risk level or likelihood of diagnosis (6, 26%). Interviews and heuristic evaluations identified more usability-related barriers and facilitators to adoption than did questionnaires and user testing studies.

Discussion

A wide range of metrics and methods are used to evaluate the usability of mobile CDSS in medical emergencies. Input of information into CDSS was predominantly manual, impeding usability. Studies employing both qualitative and quantitative methods to evaluate usability yielded more thorough results.

Conclusion

When planning CDSS projects, developers should consider multiple methods to comprehensively evaluate usability.

Effect and usability of mobile health applications for medication adherence in patients with heart failure: A systematic review

BACKGROUND

This review aimed to evaluate the effect of mobile health applications (apps) use on medication adherence in heart failure (HF) patients, and to verify their quality of use.

METHODS

We searched MEDLINE, Embase, Web of Science, Scopus, CINAHL, LILACS, Cochrane, Clinical Trial Registries, Google Scholar, ProQuest, Open Access Theses and dissertations (OATD), Annual Symposium Proceedings of the American Medical Informatics Association (AMIA).

RESULTS

Of the 836 records identified, eight studies (five randomized clinical trials (RCTs) and three observational studies) were included, totaling 484 participants, with a mean age ranging from 51.1 to 72 years and 65% were male. The apps improved medication adherence when compared with their comparators in only two RCTs (n = 111, Cohen d = 3.23, 95% confidence interval (CI) 2.66; 3.80 and n = 80, d = 1.20, 95% CI 0.71; 1.66), and two observational studies, one of them with large effect size (n = 142, Cohen d = 1.51, 95% CI 1.12; 1.90). Apps proved to be more effective interventions than their comparators allowed continuous monitoring of patients. All included studies used self-report measures to assess medication adherence behaviors and their results should be interpreted with caution, as such tools may cause social desirability bias or recall bias.

CONCLUSION

The included studies indicate a possible outcome in favor of the use of apps to improve medication adherence in HF patients, but with very low quality level of evidence. Further studies are needed to investigate the effects of the use of apps, optimal frequency and duration of its use in HF patients.

Inpatient EHR User Experience and Hospital EHR Safety Performance

IMPORTANCE

Despite the broad adoption and optimization of electronic health record (EHR) systems across the continuum of care, serious usability and safety problems persist.

OBJECTIVE

To assess whether EHR safety performance is associated with EHR frontline user experience in a national sample of hospitals.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study included all US adult hospitals that used the National Quality Forum Leapfrog Health IT Safety Measure and also used the ARCH Collaborative EHR User experience survey from January 1, 2017, to January 1, 2019. Data analysis was performed from September 2020 to November 2022.

MAIN OUTCOMES AND MEASURES

The primary outcomes were hospital performance on the Leapfrog Health IT Safety measure (overall and 10 subcomponents) and the ARCH collaborative frontline user experience scores (overall and 8 subcomponents). Ordinary least squares models with survey responses clustered by hospital were used to assess associations between the overall measures and their subcomponents.

RESULTS

There were 112 hospitals and 5689 frontline user surveys included in the study. Hospitals scored a mean of 0.673 (range, 0.297-0.973) on the Leapfrog Health IT safety measure; the mean ARCH EHR user experience score was 3.377 (range, 1 [best] to 5 [worst]). The adjusted β coefficient between the overall safety score and overall user experience score was 0.011 (95% CI, 0.006-0.016). The ARCH overall score was also significantly associated with 10 subcategory scores of the Leapfrog Health IT safety score, and the overall Leapfrog score was associated with the 8 subcategory scores of the ARCH user experience score.

CONCLUSIONS AND RELEVANCE

This cross-sectional study found a positive association between frontline user-rated EHR usability and EHR safety performance. This finding suggests that improving EHR usability, which is a current well-known pain point for EHR users, could have direct benefits in terms of improved EHR safety.

Outcome's Classification in Mobile Applications Tailored to Parents of Premature Infants: A Systematic Review

Background

Integration of healthcare services for preterm neonates at home and hospital by mobile technology is an economical and convenient intervention, which is being increasingly applied worldwide. We aimed to classify the outcomes of mobile applications tailored to parents of premature infants.

Methods

This systematic review was conducted by searching the six main databases until May 2021. Mobile applications tailored to parents of premature infants and the reported outcomes of this technology were identified and classified. Quality of screened articles checked by MMAT tool.

Results

Overall, 10703 articles were retrieved, and after eliminating the duplicated articles, 9 articles were reviewed ultimately. Identified outcomes were categorized into three groups parental, application, and neonatal outcomes. In the parental outcomes, maternal stress/stress coping, parenting self-efficacy, satisfaction, anxiety, partnership advocacy/improved parent-infant relationship, feeling of being safe, reassurance and confidence, increase awareness, as well as discharge preparedness, were identified. In the application outcomes, application usage, ease of use/user-friendly, and usability of the designed application were placed. Finally, the neonatal outcomes include health and clinical items.

Conclusion

Mobile applications can be useful in prematurity for educating pregnant mothers, managing stress and anxiety, supporting families, and preparing for discharge. Moreover, due to the coronavirus condition, providing remote services for parents is an appropriate solution to reduce the in-person visits to neonatal care centers. Development of tailored apps can promote the neonates' health and reduce their parents' stress.

Usability Study of an Electronic Medical Record From the Nurse Practitioners' Practice: A Qualitative Study Using the Think-Aloud Technique

INTRODUCTION

 Ensuring the usability of electronic medical records (EMRs) is crucial for healthcare providers to offer efficient, effective, and safe patient care. Nurse practitioners (NPs) are integral to the healthcare system and are essential in managing patient workflows. However, few studies assess NPs' perspectives on how EMR usage affects workflow and patient care quality.

METHOD

In this study, the "think-aloud technique" was utilized for usability testing. It involves observing users (NPs) as they complete their everyday tasks on the EMR while vocalizing their thoughts and emotions. This method has been proven reliable and valid through various research, such as a systematic review.

RESULTS

The EMR system used by NPs can create a heavy cognitive workload, have limited functionality, can lead to unintended errors, and may experience technical difficulties.

CONCLUSION

The EMR system used by NPs is challenging due to three main issues: high cognitive workload, limited system functionality, and technical problems. To improve the system, it is recommended to reduce the cognitive burden by customizing the user interface to fit the NPs' needs, enhancing the system's functionality by adding essential features and fixing any technical issues.

Human factors and safety analysis methods used in the design and redesign of electronic medication management systems: A systematic review

INTRODUCTION

Poorly designed electronic medication management systems (EMMS) or computerized physician order entry (CPOE) systems in hospital settings can result in usability issues and in turn, patient safety risks. As a safety science, human factors and safety analysis methods have potential to support the safe and usable design of EMMS.

OBJECTIVE

To identify and describe human factors and safety analysis methods that have been used in the design or redesign of EMMS used in hospital settings.

MATERIALS AND METHODS

A systematic review, following PRISMA guidelines, was conducted by searching online databases and relevant journals from January 2011 to May 2022. Studies were included if they described the practical application of human factors and safety analysis methods to support the design or redesign of a clinician-facing EMMS, or its components. Methods used were extracted and mapped to human centered design (HCD) activities: understanding context of use; specifying user requirements; producing design solutions; and evaluating the design.

RESULTS

Twenty-one papers met the inclusion criteria. Overall, 21 human factors and safety analysis methods were used in the design or redesign of EMMS with prototyping, usability testing, participant surveys/questionnaires and interviews the most frequent. Human factors and safety analysis methods were most frequently used to evaluate the design of a system (n = 67; 56.3%). Nineteen of 21 (90%) methods used aimed to identify usability issues and/or support iterative design; only one paper utilized a safety-oriented method and one, a mental workload assessment method.

DISCUSSION AND CONCLUSION

While the review identified 21 methods, EMMS design primarily utilized a subset of available methods, and rarely a method focused on safety. Given the high-risk nature of medication management in complex hospital environments, and the potential for harm due to poorly designed EMMS, there is significant potential to apply more safety-oriented human factors and safety analysis methods to support EMMS design.

Design and validation of a new Healthcare Systems Usability Scale (HSUS) for clinical decision support systems: a mixed-methods approach

OBJECTIVE

To develop and validate a questionnaire to assess the usability of clinical decision support systems (CDSS) and to assist in the early identification of usability issues that may impact patient safety and quality of care.

DESIGN

Mixed research methods were used to develop and validate the questionnaire. The qualitative study involved scale item development, content and face validity. Pilot testing established construct validity using factor analysis and facilitated estimates for reliability and internal consistency using the Cronbach's alpha coefficient.

SETTING

Two hospitals within a single National Health Service Trust.

PARTICIPANTS

We recruited a panel of 7 experts in usability and questionnaire writing for health purposes to test content validity; 10 participants to assess face validity and 78 participants for the pilot testing. To be eligible for this last phase, participants needed to be health professionals with at least 3 months experience using the local hospital electronic patient record system.

RESULTS

Feedback from the face and content validity phases contributed to the development and improvement of scale items. The final Healthcare Systems Usability Scale (HSUS) proved quick to complete, easy to understand and was mostly worded by potential users. Exploratory analysis revealed four factors related to patient safety, task execution, alerts or recommendations accuracy, the effects of the system on workflow and ease of system use. These separate into four subscales: patient safety and decision effectiveness (seven items), workflow integration (six items), work effectiveness (five items) and user control (four items). These factors affect the quality of care and clinician's ability to make informed and timely decisions when using CDSS. The HSUS has a very good reliability with global Cronbach's alpha 0.914 and between 0.702 and 0.926 for the four subscales.

CONCLUSION

The HSUS is a valid and reliable tool for usability testing of CDSS and early identification of usability issues that may cause medical adverse events.

Improving usability of Electronic Health Records in a UK Mental Health setting: a feasibility study

Background

Electronic Health Records (EHRs) can help clinicians to plan, document and deliver care for patients in healthcare services. When used consistently, EHRs can advance patient safety and quality, and reduce clinician’s workload. However, usability problems can make it difficult for clinicians to use EHRs effectively, which can negatively impact both healthcare professionals and patients.

Objective

To improve usability of EHRs within a mental health service in the UK.

Methods

This was a feasibility study conducted with two mental health teams. A mixed-methods approach was employed. Focus group discussions with clinicians identified existing usability problems in EHRs and changes were made to address these problems. Updated EHR assessment forms were evaluated by comparing the following measures pre and post changes: (1) usability testing to monitor time spent completing and duplicating patient information in EHRs, (2) clinician’s experience of using EHRs, and (3) proportion of completed EHR assessment forms.

Results

Usability testing with clinicians (n = 3) showed that the time taken to complete EHR assessment forms and time spent duplicating patient information decreased. Clinician’s experience of completing EHR assessment forms also significantly improved post changes compared to baseline (n = 71; p < 0.005). There was a significant increase in completion of most EHR forms by both teams after EHR usability improvements (all at p < 0.01).

Conclusions

Usability improvements to EHRs can reduce the time taken to complete forms, advance clinician’s experience and increase usage of EHRs. It is important to engage healthcare professionals in the usability improvement process of EHRs in mental health services.

Supplementary information

The online version contains supplementary material available at 10.1007/s10916-022-01832-0.

Computerized Clinical Decision Support Systems for the Early Detection of Sepsis Among Pediatric, Neonatal, and Maternal Inpatients: Scoping Review

BACKGROUND

Sepsis is a severe condition associated with extensive morbidity and mortality worldwide. Pediatric, neonatal, and maternal patients represent a considerable proportion of the sepsis burden. Identifying sepsis cases as early as possible is a key pillar of sepsis management and has prompted the development of sepsis identification rules and algorithms that are embedded in computerized clinical decision support (CCDS) systems.

OBJECTIVE

This scoping review aimed to systematically describe studies reporting on the use and evaluation of CCDS systems for the early detection of pediatric, neonatal, and maternal inpatients at risk of sepsis.

METHODS

MEDLINE, Embase, CINAHL, Cochrane, Latin American and Caribbean Health Sciences Literature (LILACS), Scopus, Web of Science, OpenGrey, ClinicalTrials.gov, and ProQuest Dissertations and Theses Global (PQDT) were searched by using a search strategy that incorporated terms for sepsis, clinical decision support, and early detection. Title, abstract, and full-text screening was performed by 2 independent reviewers, who consulted a third reviewer as needed. One reviewer performed data charting with a sample of data. This was checked by a second reviewer and via discussions with the review team, as necessary.

RESULTS

A total of 33 studies were included in this review-13 (39%) pediatric studies, 18 (55%) neonatal studies, and 2 (6%) maternal studies. All studies were published after 2011, and 27 (82%) were published from 2017 onward. The most common outcome investigated in pediatric studies was the accuracy of sepsis identification (9/13, 69%). Pediatric CCDS systems used different combinations of 18 diverse clinical criteria to detect sepsis across the 13 identified studies. In neonatal studies, 78% (14/18) of the studies investigated the Kaiser Permanente early-onset sepsis risk calculator. All studies investigated sepsis treatment and management outcomes, with 83% (15/18) reporting on antibiotics-related outcomes. Usability and cost-related outcomes were each reported in only 2 (6%) of the 31 pediatric or neonatal studies. Both studies on maternal populations were short abstracts.

CONCLUSIONS

This review found limited research investigating CCDS systems to support the early detection of sepsis among pediatric, neonatal, and maternal patients, despite the high burden of sepsis in these vulnerable populations. We have highlighted the need for a consensus definition for pediatric and neonatal sepsis and the study of usability and cost-related outcomes as critical areas for future research.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/24899.

Barriers to and Facilitators for Acceptance of Comprehensive Clinical Decision Support System-Driven Care Maps for Patients With Thoracic Trauma: Interview Study Among Health Care Providers and Nurses

BACKGROUND

Comprehensive clinical decision support (CDS) care maps can improve the delivery of care and clinical outcomes. However, they are frequently plagued by usability problems and poor user acceptance.

OBJECTIVE

This study aims to characterize factors influencing successful design and use of comprehensive CDS care maps and identify themes associated with end-user acceptance of a thoracic trauma CDS care map earlier in the process than has traditionally been done. This was a planned adaptive redesign stage of a User Acceptance and System Adaptation Design development and implementation strategy for a CDS care map. This stage was based on a previously developed prototype CDS care map guided by the Unified Theory of Acceptance and Use of Technology.

METHODS

A total of 22 multidisciplinary end users (physicians, advanced practice providers, and nurses) were identified and recruited using snowball sampling. Qualitative interviews were conducted, audio-recorded, and transcribed verbatim. Generation of prespecified codes and the interview guide was informed by the Unified Theory of Acceptance and Use of Technology constructs and investigative team experience. Interviews were blinded and double-coded. Thematic analysis of interview scripts was conducted and yielded descriptive themes about factors influencing the construction and potential use of an acceptable CDS care map.

RESULTS

A total of eight dominant themes were identified: alert fatigue (theme 1), automation (theme 2), redundancy (theme 3), minimalistic design (theme 4), evidence based (theme 5), prevent errors (theme 6), comprehensive across the spectrum of disease (theme 7), and malleability (theme 8). Themes 1 to 4 addressed factors directly affecting end users, and themes 5 to 8 addressed factors affecting patient outcomes. More experienced providers prioritized a system that is easy to use. Nurses prioritized a system that incorporated evidence into decision support. Clinicians across specialties, roles, and ages agreed that the amount of extra work generated should be minimal and that the system should help them administer optimal care efficiently.

CONCLUSIONS

End user feedback reinforces attention toward factors that improve the acceptance and use of a CDS care map for patients with thoracic trauma. Common themes focused on system complexity, the ability of the system to fit different populations and settings, and optimal care provision. Identifying these factors early in the development and implementation process may facilitate user-centered design and improve adoption.

Electronic health records, mobile health, and the challenge of improving global health

Technology continues to impact healthcare around the world. This provides great opportunities, but also risks. These risks are compounded in low-resource settings where errors in planning and implementation may be more difficult to overcome. Global Health Informatics provides lessons in both opportunities and risks by building off of general Global Health. Global Health Informatics also requires a thorough understanding of the local environment and the needs of low-resource settings. Forming effective partnerships and following the lead of local experts are necessary for sustainability; it also ensures that the priorities of the local community come first. There is an opportunity for partnerships between low-resource settings and high income areas that can provide learning opportunities to avoid the pitfalls that plague many digital health systems and learn how to properly implement technology that truly improves healthcare.

Social welfare professionals willing to participate in client information system development - Results from a large cross-sectional survey

Human-centered design methods should be implemented throughout the client information system (CIS) development process to understand social welfare professionals' needs, tasks, and contexts of use. The aim of this study was to examine Finnish social welfare professionals' experiences of participating in CIS development.A national cross-sectional web-based survey on the CIS experiences of social welfare professionals (1145 respondents) was conducted in Finland in spring 2019. This study focused on statements concerning the experiences of end users with CIS development and participation. The results are reported by professional and age groups.Half (50%) of the 1145 respondents had participated in CIS development. Half (56%) knew to whom and how to send feedback to software developers, but most (87%) indicated that changes and corrections were not made according to suggestions and quickly enough. The most preferred methods of participation were telling a person in charge of information systems development about usage problems (53%) and showing developers on site how professionals work (34%); 19% were not interested in participating.Social welfare professionals are willing to participate in CIS development, but vendors and social welfare provider organizations are underutilizing this resource. Social welfare informaticists are needed to interpret the needs of end users to software developers.

Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems

BACKGROUND AND OBJECTIVES

Electronic healthcare records have become central to patient care. Evaluation of new systems include a variety of usability evaluation methods or usability metrics (often referred to interchangeably as usability components or usability attributes). This study reviews the breadth of usability evaluation methods, metrics, and associated measurement techniques that have been reported to assess systems designed for hospital staff to assess inpatient clinical condition.

METHODS

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology, we searched Medline, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, and Open Grey from 1986 to 2019. For included studies, we recorded usability evaluation methods or usability metrics as appropriate, and any measurement techniques applied to illustrate these. We classified and described all usability evaluation methods, usability metrics, and measurement techniques. Study quality was evaluated using a modified Downs and Black checklist.

RESULTS

The search identified 1336 studies. After abstract screening, 130 full texts were reviewed. In the 51 included studies 11 distinct usability evaluation methods were identified. Within these usability evaluation methods, seven usability metrics were reported. The most common metrics were ISO9241-11 and Nielsen's components. An additional "usefulness" metric was reported in almost 40% of included studies. We identified 70 measurement techniques used to evaluate systems. Overall study quality was reflected in a mean modified Downs and Black checklist score of 6.8/10 (range 1-9) 33% studies classified as "high-quality" (scoring eight or higher), 51% studies "moderate-quality" (scoring 6-7), and the remaining 16% (scoring below five) were "low-quality."

CONCLUSION

There is little consistency within the field of electronic health record systems evaluation. This review highlights the variability within usability methods, metrics, and reporting. Standardized processes may improve evaluation and comparison electronic health record systems and improve their development and implementation.

Identifying Electronic Medication Administration Record (eMAR) Usability Issues from Patient Safety Event Reports

BACKGROUND

Improving our understanding of the association between medication errors and health information technology (health IT) usability has the potential to reduce errors and improve patient safety. This study used patient safety event reports (PSEs) to investigate the contribution of usability challenges associated with the electronic medication administration record (eMAR) to medication errors.

METHODS

Free-text descriptions of 849 medication-related PSEs selected from 2.3 million reports were analyzed. Coders identified the specific health IT components, usability challenge categories, and nuanced usability themes that contributed to each PSE. Thematic analysis was conducted to refine categorizations and identify emerging themes. Final analysis was limited to PSEs involving a contribution from eMAR, either as the point of origin or as a downstream contributor to error.

RESULTS

eMAR contributed to 473 PSEs. eMAR was the point of origin for 84 (17.8% of 473) PSEs. Usability challenge categories included Workflow support (n = 52, 11.0%) and Display/Visual clutter (n = 30, 6.3%). eMAR contributed downstream from the point of origin in 389 (82.2% of 473) PSEs, with errors stemming primarily from Pharmacy IT and computerized provider order entry (CPOE). Prominent secondary eMAR-associated usability challenges included Display/Visual clutter (n = 327, 69.1%) and Alerting (n = 32, 6.8%).

CONCLUSION

This study identified several eMAR usability challenges, through the analysis of PSEs, that contribute to medication errors. Findings highlight the critical need for improving the eMAR user interface. Improved interface design, better vendor usability testing, eMAR-focused certification testing, consideration of work system factors, and eMAR-focused usability and safety testing by health care facilities can improve eMAR technology and patient safety.

A National Survey of EMR Usability: Comparisons between medical and nursing professions in the hospital and primary care sectors in Australia and Finland

BACKGROUND

Electronic Medical Record Systems (EMRs) are now part of nursing and medical professionals daily work in the acute and primary care sectors in Australia. Usability is an important factor in their successful adoption and impacts upon clinical workflow, safety and quality, communication, and collaboration. This study replicates a significant body of work conducted by Finnish researchers applying a usability focused survey to understand medical and nursing professionals' experiences in the Australian context. As we implement EMRs across health systems, their usability and design to support clinicians to effectively deliver and document care, is essential.

METHODS

We conducted an observational study using a cross sectional survey, the National Usability-Focused HIS Scale (NuHISS) developed and validated by Finnish researchers. For this study 13 usability statements collected clinician impressions of EMRs related to technical quality, ease of use, benefits, and collaboration. We report the responses from medical and nursing professionals working in clinical practice settings in Australia, including primary care and hospital sectors in 2020.

RESULTS

Nursing and medical professionals have different experiences with EMR usability. This depends on the sector they work in and the usability feature measured. In our sample, technical quality features were more positively experienced by doctors in the primary care sector than nurses as well as ease of obtaining patient information and prevention of errors. In the hospital sector nurses experiences with EMRs were more positive with respect to support for routine task completion, learnability, ease of obtaining patient information and entry of patient data.

CONCLUSIONS

The NuHISS is a suitable tool for measuring the usability experiences of Australian clinicians and the EMRs utilised. Differences in usability experiences were noted between professional groups and sectors. A focus on the usability perspectives of clinicians when enhancing or developing EMR solutions is advocated.

Participatory Design of an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Study with Nurses and Physicians

Background: Electronic medical records (EMRs) offer a promising approach to mapping and documenting the complex information gathered in paediatric palliative care (PPC). However, if they are not well developed, poorly implemented EMRs have unintended consequences that may cause harm to patients. One approach to preventing such harm is the involvement of users in the participatory design to ensure user acceptance and patient safety. Therefore, the aim of this study is to evaluate the acceptance of a novel patient chart module (PCM) as part of an EMR from the perspective of potential users in PPC and to involve these professionals in the design process. Methods: A qualitative observational study with N = 16 PPC professionals (n = 10 nurses, n = 6 physicians) was conducted, including concurrent think aloud (CTA) and semi-structured interviews. A structured content analysis based on the Unified Theory of Acceptance and Use of Technology was applied. Results: The results can be summarized in terms of general observations, performance expectancy, effort expectancy and facilitating conditions, all of which are likely to have a positive influence on acceptance of the PCM from the user perspective in the context of PPC. Conclusions: The involvement of users in the development of EMRs is important for meeting the requirements in PPC. Further software adaptations are necessary to implement these requirements.

Optimizing Clinical Monitoring Tools to Enhance Patient Review by Pharmacists

BACKGROUND

The Clinical Monitoring List (CML) is a real-time scoring system and intervention tool used by Mayo Clinic pharmacists caring for hospitalized patients.

OBJECTIVE

The study aimed to describe the iterative development and implementation of pharmacist clinical monitoring tools within the electronic health record at a multicampus health system enterprise.

METHODS

Between October 2018 and January 2019, pharmacists across the enterprise were surveyed to determine opportunities and gaps in CML functionality. Responses were received from 39% (n = 162) of actively staffing inpatient pharmacists. Survey responses identified three main gaps in CML functionality: (1) the desire for automated checklists of tasks, (2) additional rule logic closely aligning with clinical practice guidelines, and (3) the ability to dismiss and defer rules. The failure mode and effect analysis were used to assess risk areas within the CML. To address identified gaps, two A/B testing pilots were undertaken. The first pilot analyzed the effect of updated CML rule logic on pharmacist satisfaction in the domains of automated checklists and guideline alignment. The second pilot assessed the utility of a Clinical Monitoring Navigator (CMN) functioning in conjunction with the CML to display rules with selections to dismiss or defer rules until a user-specified date. The CMN is a workspace to guide clinical end user workflows; permitting the review and actions to be completed within one screen using EHR functionality.

RESULTS

A total of 27 pharmacists across a broad range of practice specialties were selected for two separate two-week pilot tests. Upon pilot completion, participants were surveyed to assess the effect of updates on performance gaps.

CONCLUSION

Findings from the enterprise-wide survey and A/B pilot tests were used to inform final build decisions and planned enterprise-wide updated CML and CMN launch. This project serves as an example of the utility of end-user feedback and pilot testing to inform project decisions, optimize usability, and streamline build activities.

Impact of Data Entry Interface Design on Cognitive Workload, Documentation Correctness, and Documentation Efficiency

Clinical documentation serves as the legal record of patient care and used to guide clinical decision making. Inadequately designed data entry user-interfaces may result in unintended consequences that negatively impact patient safety and outcomes because inaccurate information is used to guide clinical decision making. This study utilized an electronic simulated documentation interface (i.e., artificial electronic health record) combined with eye-tracking hardware to analyze documentation correctness, documentation efficiency, and cognitive workload of anesthesia providers (N = 20) generating documentation using different computer-assisted data entry types (drop-down box, radio button, check-box, and free text with autocomplete suggestions). Our study methodology incorporating eye-tracking with electronic health record user interfaces to assess documentation correctness, efficiency, and cognitive workload can be translated to other health care provider types.

Usability of Electronic Health Record-Generated Discharge Summaries: Heuristic Evaluation

BACKGROUND

Obtaining accurate clinical information about recent acute care visits is extremely important for outpatient providers. However, documents used to communicate this information are often difficult to use. This puts patients at risk of adverse events. Elderly patients who are seen by more providers and have more care transitions are especially vulnerable.

OBJECTIVE

This study aimed to (1) identify the information about elderly patients' recent acute care visits needed to coordinate their care, (2) use this information to assess discharge summaries, and (3) provide recommendations to help improve the quality of electronic health record (EHR)-generated discharge summaries, thereby increasing patient safety.

METHODS

A literature review, clinician interviews, and a survey of outpatient providers were used to identify and categorize data needed to coordinate care for recently discharged elderly patients. Based upon those data, 2 guidelines for creating useful discharge summaries were created. The new guidelines, along with 17 previously developed medical documentation usability heuristics, were applied to assess 4 simulated elderly patient discharge summaries.

RESULTS

The initial research effort yielded a list of 29 items that should always be included in elderly patient discharge summaries and a list of 7 "helpful, but not always necessary" items. Evaluation of 4 deidentified elderly patient discharge summaries revealed that none of the documents contained all 36 necessary items; between 14 and 18 were missing. The documents each had several other issues, and they differed significantly in organization, layout, and formatting.

CONCLUSIONS

Variations in content and structure of discharge summaries in the United States make them unnecessarily difficult to use. Standardization would benefit both patients, by lowering the risk of care transition-related adverse events, and outpatient providers, by helping reduce frustration that can contribute to burnout. In the short term, acute care providers can help improve the quality of their discharge summaries by working with EHR vendors to follow recommendations based upon this study. Meanwhile, additional human factors work should determine the most effective way to organize and present information in discharge summaries, to facilitate effective standardization.

Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults

BACKGROUND

The assessment of usability is a complex process that involves several steps and procedures. It is important to standardize the evaluation and reporting of usability procedures across studies to guide researchers, facilitate comparisons across studies, and promote high-quality usability studies. The first step to standardizing is to have an overview of how usability study procedures are reported across the literature.

OBJECTIVE

This scoping review of reviews aims to synthesize the procedures reported for the different steps of the process of conducting a user-centered usability assessment of digital solutions relevant for older adults and to identify potential gaps in the present reporting of procedures. The secondary aim is to identify any principles or frameworks guiding this assessment in view of a standardized approach.

METHODS

This is a scoping review of reviews. A 5-stage scoping review methodology was used to identify and describe relevant literature published between 2009 and 2020 as follows: identify the research question, identify relevant studies, select studies for review, chart data from selected literature, and summarize and report results. The research was conducted on 5 electronic databases: PubMed, ACM Digital Library, IEEE, Scopus, and Web of Science. Reviews that met the inclusion criteria (reporting on user-centered usability evaluation procedures for any digital solution that could be relevant for older adults and were published in English) were identified, and data were extracted for further analysis regarding study evaluators, study participants, methods and techniques, tasks, and test environment.

RESULTS

A total of 3958 articles were identified. After a detailed screening, 20 reviews matched the eligibility criteria. The characteristics of the study evaluators and participants and task procedures were only briefly and differently reported. The methods and techniques used for the assessment of usability are the topics that were most commonly and comprehensively reported in the reviews, whereas the test environment was seldom and poorly characterized.

CONCLUSIONS

A lack of a detailed description of several steps of the process of assessing usability and no evidence on good practices of performing it suggests that there is a need for a consensus framework on the assessment of user-centered usability evaluation. Such a consensus would inform researchers and allow standardization of procedures, which are likely to result in improved study quality and reporting, increased sensitivity of the usability assessment, and improved comparability across studies and digital solutions. Our findings also highlight the need to investigate whether different ways of assessing usability are more sensitive than others. These findings need to be considered in light of review limitations.

Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis

BACKGROUND

Electronic health record (EHR) usability and physician task load both contribute to physician professional burnout. The association between perceived EHR usability and workload has not previously been studied at a national level. Better understanding these interactions could give further information as to the drivers of extraneous task load.

OBJECTIVE

This study aimed to determine the relationship between physician-perceived EHR usability and workload by specialty and evaluate for associations with professional burnout.

METHODS

A secondary analysis of a cross-sectional survey of US physicians from all specialties was conducted from October 2017 to March 2018. Among the 1250 physicians invited to respond to the subsurvey analyzed here, 848 (67.8%) completed it. EHR usability was assessed with the System Usability Scale (SUS; range: 0-100). Provider task load (PTL) was assessed using the mental demand, physical demand, temporal demand, and effort required subscales of the National Aeronautics and Space Administration Task Load Index (range: 0-400). Burnout was measured using the Maslach Burnout Inventory.

RESULTS

The mean scores were 46.1 (SD 22.1) for SUS and 262.5 (SD 71.7) for PTL. On multivariable analysis adjusting for age, gender, relationship status, medical specialty, practice setting, hours worked per week, and number of nights on call per week, physician-rated EHR usability was associated with PTL, with each 1-point increase in SUS score (indicating more favorable) associated with a 0.57-point decrease in PTL score (P<.001). On mediation analysis, higher SUS score was associated with lower PTL score, which was associated with lower odds of burnout.

CONCLUSIONS

A strong association was observed between EHR usability and workload among US physicians, with more favorable usability associated with less workload. Both outcomes were associated with the odds of burnout, with task load acting as a mediator between EHR usability and burnout. Improving EHR usability while decreasing task load has the potential to allow practicing physicians more working memory for medical decision making and patient communication.

A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality

Health-related virtual reality (VR) applications for patient treatment, rehabilitation, and medical professional training are on the rise. However, there is little guidance on how to select and perform usability evaluations for VR health interventions compared to the supports that exist for other digital health technologies. The purpose of this viewpoint paper is to present an introductory summary of various usability testing approaches or methods that can be used for VR applications. Along with an overview of each, a list of resources is provided for readers to obtain additionally relevant information. Six categories of VR usability evaluations are described using a previously developed classification taxonomy specific to VR environments: (1) cognitive or task walkthrough, (2) graphical evaluation, (3) post hoc questionnaires or interviews, (4) physical performance evaluation, (5) user interface evaluation, and (6) heuristic evaluation. Given the growth of VR in health care, rigorous evaluation and usability testing is crucial in the development and implementation of novel VR interventions. The approaches outlined in this paper provide a starting point for conducting usability assessments for health-related VR applications; however, there is a need to also move beyond these to adopt those from the gaming industry, where assessments for both usability and user experience are routinely conducted.

The Electronic Health Record in Ophthalmology: Usability Evaluation Tools for Health Care Professionals

Introduction

The adoption of the electronic health record (EHR) has grown rapidly in ophthalmology. However, despite its potential advantages, its implementation has often led to dissatisfaction amongst health care professionals (HCP). This can be addressed using a user centred design (UCD) which is based on the philosophy that ‘the final product should suit the users, rather than making the users suit the product’. There is often no agreed best practice on the role of HCPs in the UCD process. In this paper, we describe practical qualitative methodologies that can be used by HCPs in the design, implementation and evaluation of ophthalmology EHRs.

Methods

A review of current qualitative usability methodologies was conducted by practising ophthalmologists who are also qualified health informaticians.

Results

We identified several qualitative methodologies that could be used for EHR evaluation. These include:

Tools for user centred design: shadowing and autoethnography, semi-structured interviews and questionnaires

Tools for summative testing: card sort and reverse card sort, retrospective think aloud protocol, wireframing, screenshot testing and heat maps

Conclusion

High-yield, low-fidelity tools can be used to engage HCPs with the process of ophthalmology EHR design, implementation and evaluation. These methods can be used by HCPs without the requirement for prior training in usability science, and by clinical centres without significant technical requirements.

Characterizing and Visualizing Display and Task Fragmentation in the Electronic Health Record: Mixed Methods Design

BACKGROUND

The complexity of health care data and workflow presents challenges to the study of usability in electronic health records (EHRs). Display fragmentation refers to the distribution of relevant data across different screens or otherwise far apart, requiring complex navigation for the user's workflow. Task and information fragmentation also contribute to cognitive burden.

OBJECTIVE

This study aims to define and analyze some of the main sources of fragmentation in EHR user interfaces (UIs); discuss relevant theoretical, historical, and practical considerations; and use granular microanalytic methods and visualization techniques to help us understand the nature of fragmentation and opportunities for EHR optimization or redesign.

METHODS

Sunburst visualizations capture the EHR navigation structure, showing levels and sublevels of the navigation tree, allowing calculation of a new measure, the Display Fragmentation Index. Time belt visualizations present the sequences of subtasks and allow calculation of proportion per instance, a measure that quantifies task fragmentation. These measures can be used separately or in conjunction to compare EHRs as well as tasks and subtasks in workflows and identify opportunities for reductions in steps and fragmentation. We present an example use of the methods for comparison of 2 different EHR interfaces (commercial and composable) in which subjects apprehend the same patient case.

RESULTS

Screen transitions were substantially reduced for the composable interface (from 43 to 14), whereas clicks (including scrolling) remained similar.

CONCLUSIONS

These methods can aid in our understanding of UI needs under complex conditions and tasks to optimize EHR workflows and redesign.

The Effect of Electronic Health Record Usability Redesign on Annual Screening Rates in an Ambulatory Setting

OBJECTIVES

Improving the usability of electronic health records (EHR) continues to be a focus of clinicians, vendors, researchers, and regulatory bodies. To understand the impact of usability redesign of an existing, site-configurable feature, we evaluated the user interface (UI) used to screen for depression, alcohol and drug misuse, fall risk, and the existence of advance directive information in ambulatory settings.

METHODS

As part of a quality improvement project, based on heuristic analysis, the existing UI was redesigned. Using an iterative, user-centered design process, several usability defects were corrected. Summative usability testing was performed as part of the product development and implementation cycle. Clinical quality measures reflecting rolling 12-month rates of screening were examined over 8 months prior to the implementation of the redesigned UI and 9 months after implementation.

RESULTS

Summative usability testing demonstrated improvements in task time, error rates, and System Usability Scale scores. Interrupted time series analysis demonstrated significant improvements in all screening rates after implementation of the redesigned UI compared with the original implementation.

CONCLUSION

User-centered redesign of an existing site-specific UI may lead to significant improvements in measures of usability and quality of patient care.

Methodological Quality of Manuscripts Reporting on the Usability of Mobile Applications for Pain Assessment and Management: A Systematic Review

BACKGROUND

There has been increasing use of mobile mHealth applications, including pain assessment and pain self-management apps. The usability of mHealth applications has vital importance as it affects the quality of apps. Thus, usability assessment with methodological rigor is essential to minimize errors and undesirable consequences, as well as to increase user acceptance.

OBJECTIVE

this study aimed to synthesize and evaluate existing studies on the assessment of the usability of pain-related apps using a newly developed scale.

METHODS

an electronic search was conducted in several databases, combining relevant keywords. Then titles and abstracts were screened against inclusion and exclusion criteria. The eligible studies were retrieved and independently screened for inclusion by two authors. Disagreements were resolved by discussion until consensus was reached.

RESULTS

a total of 31 articles were eligible for inclusion. Quality assessment revealed that most manuscripts did not assess usability using valid instruments or triangulation of methods of usability assessment. Most manuscripts also failed to assess the three domains of usability (effectiveness, efficiency and satisfaction).

CONCLUSIONS

future studies should consider existing guidelines on usability assessment design, development and assessment of pain-related apps.

How well is the electronic health record supporting the clinical tasks of hospital physicians? A survey of physicians at three Norwegian hospitals

Background

The electronic health record is expected to improve the quality and efficiency of health care. Many novel functionalities have been introduced in order to improve medical decision making and communication between health care personnel. There is however limited evidence on whether these new functionalities are useful. The aim of our study was to investigate how well the electronic health record system supports physicians in performing basic clinical tasks.

Methods

Physicians of three prominent Norwegian hospitals participated in the survey. They were asked, in an online questionnaire, how well the hospital’s electronic health record system DIPS supported 49 clinical tasks as well as how satisfied they were with the system in general, including the technical performance. Two hundred and eight of 402 physicians (52%) submitted a completely answered questionnaire.

Results

Seventy-two percent of the physicians had their work interrupted or delayed because the electronic health record hangs or crashes at least once a week, while 22% had experienced this problem daily. Fifty-three percent of the physicians indicated that the electronic health record is cumbersome to use and adds to their workload. The majority of physicians were satisfied with managing tests, e.g., requesting laboratory tests, reading test results and managing radiological investigations and electrocardiograms. Physicians were less satisfied with managing referrals. There was high satisfaction with some of the decision support functionalities available for prescribing drugs. This includes drug interaction alerts and drug allergy warnings, which are displayed automatically. However, physicians were less satisfied with other aspects of prescribing drugs, including getting an overview of the ongoing drug therapy.

Conclusions

In the survey physicians asked for improvements of certain electronic health record functionalities like medication, clinical workflow support including planning and better overviews. In addition, there is apparently a need to focus on system stability, number of logins, reliability and better instructions on available electronic health record features. Considerable development is needed in current electronic health record systems to improve usefulness and satisfaction.

A Scale to Assess the Methodological Quality of Studies Assessing Usability of Electronic Health Products and Services: Delphi Study Followed by Validity and Reliability Testing

BACKGROUND

The usability of electronic health (eHealth) and mobile health apps is of paramount importance as it impacts the quality of care. Methodological quality assessment is a common practice in the field of health for different designs and types of studies. However, we were unable to find a scale to assess the methodological quality of studies on the usability of eHealth products or services.

OBJECTIVE

This study aimed to develop a scale to assess the methodological quality of studies assessing usability of mobile apps and to perform a preliminary analysis of of the scale's feasibility, reliability, and construct validity on studies assessing usability of mobile apps, measuring aspects of physical activity.

METHODS

A 3-round Delphi panel was used to generate a pool of items considered important when assessing the quality of studies on the usability of mobile apps. These items were used to write the scale and the guide to assist its use. The scale was then used to assess the quality of studies on usability of mobile apps for physical activity, and it assessed in terms of feasibility, interrater reliability, and construct validity.

RESULTS

A total of 25 experts participated in the Delphi panel, and a 15-item scale was developed. This scale was shown to be feasible (time of application mean 13.10 [SD 2.59] min), reliable (intraclass correlation coefficient=0.81; 95% CI 0.55-0.93), and able to discriminate between low- and high-quality studies (high quality: mean 9.22 [SD 0.36]; low quality: mean 6.86 [SD 0.80]; P=.01).

CONCLUSIONS

The scale that was developed can be used both to assess the methodological quality of usability studies and to inform its planning.

Evaluation of the electronic health record system in maternal and child health centers of Marie Stopes International Ethiopia

Background: The Health Information Management Systems Society (HIMSS) defines an Electronic Health Record (EHR) as a longitudinal electronic record of patient health information generated by one or more encounters in any care delivery setting”. EHR systems improve quality and adaptability of care, contributing to patient safety. The system is also efficient and provides real-time administration to patients. Driven by the need to facilitate clinical and administrative process, and to reduce medical errors, Marie Stopes international Ethiopia (MSIE) implemented EHR system in its maternal and child health (MCH) centers. This study sought to evaluate the EHR system in six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers) of these MCH centers in terms of ease of performing tasks and satisfaction with the system. Methods: A cross-sectional method with formative evaluation was used to address the objective. A sample size of 54 were used for the investigation. The participants were from six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers) within three MCH centers. Data was collected using structured, self-administered questionnaires and interviews and analyzed using SPSS 20. Results: In total, 83% of respondents found that performing tasks using EHR made tasks easier; the lowest score was from Pharmacy department (48%). The overall satisfaction with EHR was 87%. The lowest score is from the indicator “the system meets my needs” (80%). Conclusions: Staffs in all department reported the EHR system as “very good” for both ease of performing their daily tasks and satisfaction with the system. In addition, the study revealed that ease of performing tasks and satisfaction with the system were varied by department and specific task within the department. It is essential to know the needs and requirements of each department before implementation of the system and user feedback for long lasting uptake and impact.

Critical care information display approaches and design frameworks: A systematic review and meta-analysis

OBJECTIVE

To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians.

METHODS

We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author.

RESULTS

Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible.

CONCLUSION

Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.

Usability Evaluation of Visual Representation Formats for Emergency Department Records

BACKGROUND

Integration of electronic information is a challenge for multitasking emergency providers, with implications for patient safety. Visual representations can assist sense-making of complex data sets; however, benefit and acceptability in emergency care is unproven.

OBJECTIVES

This article evaluates visually focused alternatives to lists or tabular formats, to better understand possible usability in Emergency Department Information System (EDIS).

METHODS

A counterbalanced, repeated-measures experiment, satisfaction surveys, and narrative content analysis was conducted remotely by Web platform. Participants were 37 American emergency physicians; they completed 16 clinical cases comparing 4 visual designs to the control formats from a commercially available EDIS. They then evaluated two additional chart overview representations without controls.

RESULTS

Visual designs provided benefit in several areas compared to controls. Task correctness (90% to 76%; p = 0.003) and completion time (median: 49-74 seconds; p < 0.001) were superior for a medication history timeline with class and schedule highlighting. Completion time (median: 45-60 seconds; p = 0.03) was superior for a past medical history design, using pertinent diagnosis codes in highlighting rules. Less mental effort was reported for visual allergy (p = 0.04), past medical history (p < 0.001), and medication timeline (p < 0.001) designs. Most of the participants agreed with statements of likeability, preference, and benefit for visual designs; nonetheless, contrary opinions were seen, and more complex designs were viewed less favorably.

CONCLUSION

Physician performance with visual representations of clinical data can in some cases exceed standard formats, even in absence of training. Highlighting of priority clinical categories was rated easier-to-use on average than unhighlighted controls. Perceived complexity of timeline representations can limit desirability for a subset of users, despite potential benefit.

Identifying Electronic Health Record Usability And Safety Challenges In Pediatric Settings

Pediatric populations are uniquely vulnerable to the usability and safety challenges of electronic health records (EHRs), particularly those related to medication, yet little is known about the specific issues contributing to hazards. To understand specific usability issues and medication errors in the care of children, we analyzed 9,000 patient safety reports, made in the period 2012-17, from three different health care institutions that were likely related to EHR use. Of the 9,000 reports, 3,243 (36 percent) had a usability issue that contributed to the medication event, and 609 (18.8 percent) of the 3,243 might have resulted in patient harm. The general pattern of usability challenges and medication errors were the same across the three sites. The most common usability challenges were associated with system feedback and the visual display. The most common medication error was improper dosing.

A Daily Hospital Progress Note that Increases Physician Usability of the Electronic Health Record by Facilitating a Problem-Oriented Approach to the Patient and Reducing Physician Clerical Burden

We suggest changes in the electronic health record (EHR) in hospitalized patients to increase EHR usability by optimizing the physician's ability to approach the patient in a problem-oriented fashion and by reducing physician data entry and chart navigation. The framework for these changes is a Physician's Daily Hospital Progress Note organized into 3 sections: Subjective, Objective, and a combined Assessment and Plan section, subdivided by problem titles. The EHR would consolidate information for each problem by: 1) juxtaposing to each problem title relevant medications, key durable results, and limitations; 2) entering in the running lists under Assessment and Plan the most relevant information for that day, including abbreviated versions of relevant reports; and 3) generating a flow sheet in a problem's progress note for any key results tracked daily. To reduce physician EHR navigation, the EHR would place in the Objective section abbreviated versions of notes of other physicians, nurses, and allied health professionals as well as recent orders. The physician would enter only the analysis and plan and new information not included in the EHR. The consolidation of information for each problem would facilitate physician communication at points of transition of care including generation of a problem-oriented discharge summary.

Safeuristics! Do Heuristic Evaluation Violation Severity Ratings Correlate with Patient Safety Severity Ratings for a Native Electronic Health Record Mobile Application?

OBJECTIVE

Usability of electronic health records (EHRs) remains challenging, and poor EHR design has patient safety implications. Heuristic evaluation detects usability issues that can be classified by severity. The National Institute of Standards and Technology provides a safety scale for EHR usability. Our objectives were to investigate the relationship between heuristic severity ratings and safety scale ratings in an effort to analyze EHR safety.

MATERIALS AND METHODS

Heuristic evaluation was conducted on seven common mobile EHR tasks, revealing 58 heuristic violations and 28 unique usability issues. Each usability issue was independently scored for severity by trained hospitalists and a Human Factors researcher and for safety severity by two physician informaticists and two clinical safety professionals.

RESULTS

Results demonstrated a positive correlation between heuristic severity and safety severity ratings. Regression analysis demonstrated that 49% of safety risk variability by clinical safety professionals (r = 0.70; n = 28) and 42% of safety risk variability by clinical informatics specialists (r = 0.65; n = 28) was explained by usability severity scoring of problems outlined by heuristic evaluation. Higher severity ratings of the usability issues were associated with increased perceptions of patient safety risk.

DISCUSSION

This study demonstrated the use of heuristic evaluation as a technique to quickly identify usability problems in an EHR that could lead to safety issues. Detection of higher severity ratings could help prioritize failures in EHR design that more urgently require design changes. This approach is a cost-effective technique for improving usability while impacting patient safety.

CONCLUSION

Results from this study demonstrate the efficacy of the heuristic evaluation technique to identify usability problems that impact safety of the EHR. Also, the use of interdisciplinary teams for evaluation should be considered for severity assessment.

Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: A case study and beginning method validation

INTRODUCTION

Self-management of chronic diseases using mobile health (mHealth) systems and applications is becoming common. Current evaluation methods such as formal usability testing can be very costly and time-consuming; others may be more efficient but lack a user focus. We propose an enhanced cognitive walkthrough (CW) method, the user-centered CW (UC-CW), to address identified deficiencies in the original technique and perform a beginning validation with think aloud protocol (TA) to assess its effectiveness, efficiency and user acceptance in a case study with diabetes patient users on a mHealth self-management application.

MATERIALS AND METHODS

A total of 12 diabetes patients at University of Utah Health, USA, were divided into UC-CW and think aloud (TA) groups. The UC-CW method included: making the user the main evaluator for detecting usability problems, having a dual domain facilitator, and using three other improved processes: validated task development, higher level tasks and a streamlined evaluation process. Users interacted with the same mHealth application for both methods. Post-evaluation assessments included the NASA RTLX instrument and a set of brief interview questions.

RESULTS

Participants had similar demographic characteristics. A total of 26 usability problems were identified with the UC-CW and 20 with TA. Both methods produced similar ratings: severity across all views (UC-CW = 2.7 and TA = 2.6), numbers of problems in the same views (Main View [UC-CW = 11, TA = 10], Carbohydrate Entry View [UC-CW = 4, TA = 3] and List View [UC-CW = 3, TA = 3]) with similar heuristic violations (Match Between the System and Real World [UC-CW = 19, TA = 16], Consistency and Standards [UC-CW = 17, TA = 15], and Recognition Rather than Recall [UC-CW = 13, TA = 10]). Both methods converged on eight usability problems, but the UC-CW group detected five critical issues while the TA group identified two. The UC-CW group identified needed personalized features for patients' disease needs not identified with TA. UC-CW was more efficient on average time per identified usability problem and on the total evaluation process with patients. NASA RTLX scores indicated that participants experienced the UC-CW half as cognitively demanding. Common themes from interviews indicated the UC-CW as enjoyable and easy to perform while TA was considered somewhat awkward and more cognitively challenging.

CONCLUSIONS

UC-CW was effective for finding severe, recurring usability problems and it highlighted the need for personalized user features. The method was also efficient and had high user acceptance. These results indicate UC-CW's utility and user acceptance in evaluating a mHealth self-management application. It provides an additional usability evaluation technique for researchers.

Inpatient electronic health record maintenance from 2010 to 2015

OBJECTIVES

To describe the scale and scope of inpatient electronic health record (EHR) maintenance following initial implementation.

STUDY DESIGN

A retrospective study reviewing EHR change documentation within an integrated healthcare delivery system that has 21 hospitals.

METHODS

Between 2010 and 2015, we identified and categorized all significant changes made to the inpatient EHR, as documented within monthly EHR communication updates. We categorized EHR changes as updates to existing functionality or upgrades to new functionality. We grouped changes within larger functional domains as orders, alerts and customization, surgical and emergency department (ED), data review, reports and health information management, and other. We also identified the clinical areas and user roles targeted by these changes.

RESULTS

Over a 6-year period, 5551 unique changes were made to the inpatient EHR, with a median of 72 changes per month. Changes most frequently targeted orders (44.7% of 2190 change documents) and order sets (29.9% of documents). In total, changes affected 135 EHR functions. A total of 151 unique user roles were affected by these changes, with the most frequent roles including nurses (30.6%), physicians (26.6%), and other clinical staff (22.7%). The clinical areas most targeted by changes included surgical areas and the ED.

CONCLUSIONS

Over 6 years, EHR maintenance for clinical functionality was substantial and varied with pervasive impacts, requiring persistent attention, diverse expertise, and interdisciplinary collaboration.

Safety and Usability Guidelines of Clinical Information Systems Integrating Clinical Workflow: A Systematic Review

Objectives

The usability of clinical information systems (CISs) is known to be an essential consideration in ensuring patient safety as well as integrating clinical flow. This study aimed to determine how usability and safety guidelines of CIS consider clinical workflow through a systematic review in terms of the target systems, methodology, and guideline components of relevant articles.

Methods

A literature search was conducted for articles published from 2000 to 2015 in PubMed, Cochrane, EMBASE, Web of Science, and CINAHL. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement method was employed. Articles containing recommendations, principles, and evaluation items for CIS usability and safety were included. The selected articles were classified according to article type, methodology, and target systems. Taking clinical workflow into consideration, the components of guidelines were extracted and classified.

Results

A total of 7,401 articles were identified by keyword search. From the 76 articles remaining after abstract screening, 15 were selected through full-text review. Literature review (n = 7) was the most common methodology, followed by expert opinions (n = 6). Computerized physician order entry (n = 6) was the most frequent system. Four articles considered the entire process of clinical tasks, and two articles considered the principles of the entire process of user interface affecting clinical workflow. Only two articles performed heuristic evaluations of CISs.

Conclusions

The usability and safety guidelines of CISs need improvement in guideline development methodology and with consideration of clinical workflow.

Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method

Background and aim

Obstetrics and gynecology information system is a critical component of the HIS in social security organization health centers. The objective of this study was to evaluate the usability of this system using the cognitive walkthrough method. Also, the present study provided a detailed formal description of how the cognitive usability evaluation can be applied and reported for a health care information system.

Methods

This study was conducted at the Mashhad University of Medical Sciences' usability lab from March 2016 to June 2017. A two-phase approach was used to conduct the cognitive walkthrough evaluation: preparatory and evaluation. The preparation was done in three stages: first, we investigated users' capabilities and background knowledge through a semi-structured interview. Second, the evaluation scenario was developed based on the most common tasks in routine workflow of users. Finally, each task was broken down into sequences of actions. In the evaluation phase, three usability experts independently assessed each action using a four-item checklist. Problems were categorized thematically and were reported from three different perspectives: Question-based, Task-based, and Evaluator-based. The data were then analyzed to understand the contribution of each task, along with its mean severity score.

Results

Evaluators' responses were compared and any conflict was resolved in an expert panel. A total of 116 usability problems were identified based on the consensus of the evaluators. Inadequate system feedback was found to be the main source of 43% of the problems, and resulted in users confusion.

Conclusion

Since the system was evaluated in its pilot implementation phase, there was an opportunity to prevent future potential usability problems. The use of a mixed quantitative and qualitative approach in this usability study provided a more comprehensive perspective of the system problems. This study provided a detailed description of conducting CW usability evaluation which can be used as a practical guide for future studies.

Electronic Health Record Usability Issues and Potential Contribution to Patient Harm

This study analyzed patient safety reports in and near Pennsylvania from 2013 through 2016 to identify those that contained explicit language associating possible patient harm with an electronic health record usability issue.

A six-year repeated evaluation of computerized clinical decision support system user acceptability

OBJECTIVE

Long-term acceptability among computerized clinical decision support system (CDSS) users in pediatrics is unknown. We examine user acceptance patterns over six years of our continuous computerized CDSS integration and updates.

MATERIALS AND METHODS

Users of Child Health Improvement through Computer Automation (CHICA), a CDSS integrated into clinical workflows and used in several urban pediatric community clinics, completed annual surveys including 11 questions covering user acceptability. We compared responses across years within a single healthcare system and between two healthcare systems. We used logistic regression to assess the odds of a favorable response to each question by survey year, clinic role, part-time status, and frequency of CHICA use.

RESULTS

Data came from 380 completed surveys between 2011 and 2016. Responses were significantly more favorable for all but one measure by 2016 (OR range 2.90-12.17, all p < 0.01). Increasing system maturity was associated with improved perceived function of CHICA (OR range 4.24-7.58, p < 0.03). User familiarity was positively associated with perceived CDSS function (OR range 3.44-8.17, p < 0.05) and usability (OR range 9.71-15.89, p < 0.01) opinions.

CONCLUSION

We present a long-term, repeated follow-up of user acceptability of a CDSS. Favorable opinions of the CDSS were more likely in frequent users, physicians and advanced practitioners, and full-time workers. CHICA acceptability increased as it matured and users become more familiar with it. System quality improvement, user support, and patience are important in achieving wide-ranging, sustainable acceptance of CDSS.

Healthcare information systems: the cognitive challenge

BACKGROUND

Healthcare work is, to a considerable extent, cognitive. Subsequently, the analysis and the design of supporting technology must be sensitive to the cognitive and adaptive demands of the work and to the cognitive strategies employed by healthcare practitioners. Despite the vital role that cognition plays in healthcare work, current technocentric design approaches for healthcare technology do not account for it, failing to observe it during analysis and failing to develop support for it during design.

MAIN BODY

By review and analysis of case studies, we show that healthcare systems developed without input from cognitive analysis and cognitive design fail to take account of important healthcare work processes and workflows. In contrast, systems developed with a cognitively-focused design strategy demonstrate how it is possible to introduce technology that supports and enhances the work strategies of those engaged in patient care.

CONCLUSION

Significant problems emerge when technological support systems are developed without any serious and comprehensive attempt to understand the cognitive capabilities and skills deployed by those involved in patient care. In contrast, significant benefits accrue from taking full account of those cognitive capabilities and skills. Subsequently, the design and development of supporting technology must be sensitive to the cognitive demands of the work and the cognitive strategies employed by healthcare practitioners.

Clinical information extraction applications: A literature review

BACKGROUND

With the rapid adoption of electronic health records (EHRs), it is desirable to harvest information and knowledge from EHRs to support automated systems at the point of care and to enable secondary use of EHRs for clinical and translational research. One critical component used to facilitate the secondary use of EHR data is the information extraction (IE) task, which automatically extracts and encodes clinical information from text.

OBJECTIVES

In this literature review, we present a review of recent published research on clinical information extraction (IE) applications.

METHODS

A literature search was conducted for articles published from January 2009 to September 2016 based on Ovid MEDLINE In-Process & Other Non-Indexed Citations, Ovid MEDLINE, Ovid EMBASE, Scopus, Web of Science, and ACM Digital Library.

RESULTS

A total of 1917 publications were identified for title and abstract screening. Of these publications, 263 articles were selected and discussed in this review in terms of publication venues and data sources, clinical IE tools, methods, and applications in the areas of disease- and drug-related studies, and clinical workflow optimizations.

CONCLUSIONS

Clinical IE has been used for a wide range of applications, however, there is a considerable gap between clinical studies using EHR data and studies using clinical IE. This study enabled us to gain a more concrete understanding of the gap and to provide potential solutions to bridge this gap.