Task-specific usability requirements of electronic medical records systems: Lessons learned from a national survey of end-users

Comparison of usability evaluation methods for a health information system: heuristic evaluation versus cognitive walkthrough method

Background

There are differences of opinion regarding the selection of the most practical usability evaluation method among different methods. The present study aimed to compare two expert-based evaluation methods in order to assess a nursing module as the most widely used module of a Hospital Information System (HIS).

Methods

Five independent evaluators used the Heuristic Evaluation (HE) and Cognitive Walkthrough (CW) methods to evaluate the nursing module of Shafa HIS. In this regard, the number and severity of the recognized problems according to the usability attributes were compared using two evaluation methods.

Results

The HE and CW evaluation methods resulted in the identification of 104 and 24 unique problems, respectively, of which 33.3% of recognized problems in the CW evaluation method overlapped with the HE method. The average severity of the recognized problems was considered to be minor (2.34) in the HE method and major (2.77) in the CW evaluation method. There was a significant difference in terms of the total number and average severity of the recognized problems by these methods (P < 0.001). Based on the usability attribute, the HE method identified a larger number of problems concerning all usability attributes, and a significant difference was observed in terms of the number of recognized problems in both methods for all attributes except ‘memorability’. Also, there was a significant difference between the two methods based on the average severity of recognized problems only in terms of ‘learnability’.

Conclusion

The HE method identified more problems with lower average severity while the CW was able to recognize fewer problems with higher average severity. Regarding the evaluation goal, the HE method was able to be used to improve the effectiveness and satisfaction of the HIS. Furthermore, the CW evaluation method is recommended to identify usability problems with the highest average severity, especially in terms of ‘learnability’.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01905-7.

Identification and Classification of Usability Problems in a Nursing Information System: A Heuristic Evaluation

Nursing information system is a system that nurses are required to use in order to perform their daily activities. Poor user interface design of nursing information system can cause problems in nurses' interaction with the system. This research aimed to evaluate the usability of nursing information system. During this study, five evaluators examined the nursing information system with the heuristic evaluation method and by using the checklist of Nielsen usability principles. Then, the identified problems were categorized into 10 Nielsen usability principles, and the severity of the problems was determined. They also attributed each problem to one of usability attributes. A total of 104 unique problems were identified. Most of the problems were related to the principle of "consistency and standards." More than a third of the identified problems were classed as major and catastrophe, with the highest severity in the average problems of "help and documentation" and "error prevention." Most of the identified problems were attributed to effectiveness and satisfaction. Because a significant number of identified usability problems in nursing information system were major and catastrophe and were attributed to effectiveness and satisfaction, being able to remedy the problems could improve nurses' interaction with the system user interface and increase satisfaction and effectiveness of nurses.

A National Survey of EMR Usability: Comparisons between medical and nursing professions in the hospital and primary care sectors in Australia and Finland

BACKGROUND

Electronic Medical Record Systems (EMRs) are now part of nursing and medical professionals daily work in the acute and primary care sectors in Australia. Usability is an important factor in their successful adoption and impacts upon clinical workflow, safety and quality, communication, and collaboration. This study replicates a significant body of work conducted by Finnish researchers applying a usability focused survey to understand medical and nursing professionals' experiences in the Australian context. As we implement EMRs across health systems, their usability and design to support clinicians to effectively deliver and document care, is essential.

METHODS

We conducted an observational study using a cross sectional survey, the National Usability-Focused HIS Scale (NuHISS) developed and validated by Finnish researchers. For this study 13 usability statements collected clinician impressions of EMRs related to technical quality, ease of use, benefits, and collaboration. We report the responses from medical and nursing professionals working in clinical practice settings in Australia, including primary care and hospital sectors in 2020.

RESULTS

Nursing and medical professionals have different experiences with EMR usability. This depends on the sector they work in and the usability feature measured. In our sample, technical quality features were more positively experienced by doctors in the primary care sector than nurses as well as ease of obtaining patient information and prevention of errors. In the hospital sector nurses experiences with EMRs were more positive with respect to support for routine task completion, learnability, ease of obtaining patient information and entry of patient data.

CONCLUSIONS

The NuHISS is a suitable tool for measuring the usability experiences of Australian clinicians and the EMRs utilised. Differences in usability experiences were noted between professional groups and sectors. A focus on the usability perspectives of clinicians when enhancing or developing EMR solutions is advocated.

Investigation of usability problems of electronic medical record systems in the emergency department

BACKGROUND

Despite the benefits of using electronic medical record (EMR) systems, existing studies show that many healthcare providers are uncertain regarding their usability. The usability issues of these systems decrease their efficiency, discourage clinicians, and cause dissatisfaction among patients, which may result in safety risks and harm.

OBJECTIVE

The aim of this study was to collect and analyze EMR system usability problems from actual users. Practical user interface guidelines were presented based on the medical practices of these users.

METHODS

Employing an online questionnaire with a seven-point Likert scale, usability issues of EMR systems were collected from 200 emergency department healthcare providers (103 physicians (medical doctors) and 97 nurses) from South Korea.

RESULTS

The most common usability problem among the physicians and nurses was generating in-patient selection. This pertained to the difficulty in finding the required information on-screen because of poor visibility and a lack of distinctiveness.

CONCLUSIONS

The major problems of EMR systems and their causes were identified. It is recommended that intensive visual enhancement of EMR system interfaces should be implemented to support user tasks. By providing a better understanding of the current usability problems among medical practitioners, the results of this study can be useful for developing EMR systems with increased effectiveness and efficiency.

Investigating the Impact of Usability on Energy Efficiency of Web-based Personal Health Records

Usability plays an important role in eHealth applications for their widespread adoption. These software systems have been studied in depth in the literature from this perspective. However, the energy consumption of information systems in the eHealth domain lacks comprehensive research. In this paper, the relationship between the energy consumption of the main components of a PC and the usability evaluations of graphical user interfaces (GUI) were studied in client applications. For this purpose, personal health records (PHR) were used as a case study. A set of 4 web-based PHRs were evaluated with the performance of 20 common tasks in the aforementioned systems. The usability evaluations were carried out by 4 experts. A total of 8 heuristics particularly designed for electronic health record systems and based on the Nielsen usability heuristics were employed. The instantaneous energy values of the power supply and the monitor were collected with an energy consumption measuring equipment. As a result, the following significant correlations were found. In NoMoreClipboard between the Memory heuristic and the power supply (r_s = 0.495, p = 0.044), and the Minimalist heuristic and the monitor (r_s = - 0.513, p = 0.035). In PatientsLikeMe between the Minimalist heuristic and the power supply (r_s = - 0.479, p = 0.083), as well as in the Error heuristic and the power supply (r_s = - 0.638, p = 0.014). Finally, in Health Companion between the Match heuristic and the power supply (r_s = - 0.481, p = 0.043), and the Error heuristic and the power supply again (r_s = - 0.602, p = 0.008). Apart from that, the tasks were divided into those with excellent and fair usability. Those tasks with high usability scores were evaluated to detect whether they also had low energy consumption during task performance. Significant differences were found in PatientsLikeMe with respect to the Error heuristic and the power supply (212.39 ± 18.85W). Significant differences were also found in Health Companion with respect to the Match heuristic and the power supply (199.91 ± 7.19W). The results were discussed to reveal the fundamentals of how implementing usability can impact on energy costs when running client applications.

Usability evaluation of a nursing information system by applying cognitive walkthrough method

BACKGROUND AND OBJECTIVE

The user interface usability of the nursing information system (NIS) should be such that the nurses can learn and interact with it easily and quickly. Therefore, it is necessary to identify and solve the usability problems of these systems. The present study aimed to evaluate the usability of a NIS using the cognitive walkthrough (CW) evaluation method.

METHODS

Based on five selected scenarios, five evaluators evaluated the NIS in Shafa Hospital Information System. After identifying the problems, the evaluators assigned each problem to one of the usability attributes. The severity of each identified problem was determined by the evaluators and five real users of the system.

RESULTS

In total, 24 unique problems were identified. The average severity of the problems was determined by the evaluators (2.77) and the actual users (2.82) in the "major problem" category. The highest number of problems were assigned to the scenarios 3 and 2 with 15 and 14 problems, respectively. The highest average severity in terms of evaluations and actual users was related to the scenario 5 (3.06 and 2.94, respectively), which was in the "major problem" category. The highest number of problems were associated with learnability (8 problems) and efficiency (6 problems).

CONCLUSION

Since most of the nurses do not have enough time for learning the system, and given that the selected scenarios for evaluation were based on the most frequent daily tasks that nurses performed using the NIS; The correction of usability problems in the scenarios 3 and 2 that had the highest number of problems and the scenario 5 that had the highest average severity and execution time; it can reduce the nurses' cognitive load and the learning time of the system and increase the efficiency of nurses.

A Study on the Relationship between Usability of GUIs and Power Consumption of a PC: The Case of PHRs

Usability is key to achieve quality in software products. The client applications with a high score in usability might impact on the power consumption when they are run in a PC. For this reason, energy savings turn to be critical in green software systems. In this paper the relationship between the usability evaluations of the GUIs and the power consumption measurements of the main components of a PC were analysed. A set of 5 web-based personal health records (PHRs) were selected as a case study. The usability assessment was performed by an expert, employing the 14 principles of design by Alan Dix as heuristics. They were scored on a Likert scale after performing a collection of common tasks in the PHRs. At the same time, an equipment to measure the energy consumption of hard disk drive, graphics card, processor, monitor and power supply was used. Spearman’s index was studied for the correlations between the usability assessments and the power consumption measurements. As a results, some weak relationships were found. A total of 5 usability heuristics were observed to may influence energy consumption when they were considered in the implementation of the PHRs. These heuristics were the following ones: consistency, task migratability, observability, recoverability and responsiveness. Based on the results, the usability principles of design cannot always be related to lower energy consumption. Future research should focus on the tradeoffs between usability and power consumption of client applications when they are used in a computer.

Technical requirements framework of hospital information systems: design and evaluation

BACKGROUND

Implementing the health information system (HIS) is more complex and costly than implementing other information systems. The present study was conducted to design and evaluate technical requirements for the HIS.

METHODS

The present study was conducted in 2016 by determining technical requirements for the HIS using the Delphi technique and then evaluating this system using a checklist based on the approved requirements.

RESULTS

The first part of the study designed a 73-item final list of technical requirements for the HIS in four domains, i.e. communication service, system architecture, security service and system response time. The evaluation results obtained in the second part showed that communication service was met in 63.8% of the HIS programs, system architecture in 65.5%, security service in 72.4% and system response time in 76.3%.

CONCLUSIONS

A technical evaluation tool was designed and used to select and evaluate the HIS. The evaluation results suggested the study HIS was poorer in terms of communication service and system architecture than in the other two dimensions.

Review of the Delphi method in library and information science research

Purpose

This article presents an introduction to the Delphi method and review of Delphi studies published in the literature of library and information science (LIS).

Design/methodology/approach

A review of Delphi studies published between the years of 1971 and 2019 is performed, using studies retrieved from the Library and Information Science Source database. A total of 122 articles were retrieved and evaluated based on the population studied, means of identifying experts, number of participants for each study round, type of Delphi, and type of findings.

Findings

General librarians (any type), academic librarians, and information science researchers are the most common populations in LIS Delphi studies. On average (middle 50 percent of studies), 14–36 experts are used in the first round of LIS Delphi studies (median n = 23). Employment in a specific role and publications in scholarly journals are the most common means of identifying experts. Variants of the e-Delphi (online survey/email) method are increasingly common, particularly in LIS Delphi studies that focus on general information science, rather than library, topics. Though LIS Delphi studies are relatively few in number, they have a consistent record of being published in some of the most prestigious LIS journals.

Originality/value

This paper provides an introduction to the Delphi method for LIS research and presents an overview of existing literature in LIS that utilizes the research method. No overview of this extent exists in the LIS literature, and, thus, this paper may serve as an important information source about the method for LIS researchers.

Usability Evaluation of Three Admission and Medical Records Subsystems Integrated into Nationwide Hospital Information Systems: Heuristic Evaluation

Introduction:

Usability is one of the quality criteria for information systems and its weakness is one of the main barriers to the adoption of these systems. The purpose of this study was to evaluate the usability of admission and medical records module of three widely used hospital information systems (HISs).

Methods:

In this descriptive study the usability of admission and medical records module of three HISs (HIS1, HIS2, and HIS3) was evaluated using heuristic evaluation method. For each HIS, three expert users of the same system assessed the user interface independently, completed a usability evaluation checklist, and rated severity of each identified problem. The checklist was based on Nielsen’s heuristics. For each HIS, three heuristics that have the highest and lowest problem rates and greatest severity of problems were categorized into three separate groups. The results were analyzed using descriptive statistics.

Results:

Although HIS1 and HIS2 were used in more hospitals than HIS3, the results showed that the usability problem rates of them were significantly higher than HIS3. The heuristics of “help and documentation”, “flexibility and efficiency of use”, and “visibility of system status” in the three HISs were categorized into the “highest rate of problems”, “lowest rate of problems”, and “highest severity of problems” groups, respectively. The heuristics of “diagnose and recover from errors”, “error prevention”, and “help and documentation” in HIS1 and HIS2 were categorized into the “highest rate of problems” group.

Conclusions:

The results of this study and previous studies show that the most common usability problems with HISs are related to heuristics of “help and documentation”, “error prevention”, and “help users recognize, diagnose and recover from errors.” Also, the large number of hospitals using one HIS does not demonstrate its high usability to others.