Complementary methods of system usability evaluation: Surveys and observations during software design and development cycles

A usability evaluation of the perceived user friendliness, accessibility, and inclusiveness of a personalized digital care pathway tool

OBJECTIVE

This study aimed to acquire insight into the perceived user friendliness, accessibility and inclusiveness of a personalized digital care pathway.

MATERIALS & METHODS

Usability of the tool was tested in an experimental setting. Mixed methods data collection consisted of scenario-based eye tracking tests in a web- or mobile-based prototype of the tool, followed by a questionnaire assessing user friendliness (System Usability Scale; SUS) and a structured interview. Inclusiveness was assessed by subgroup comparisons based on language proficiency, age and education level. Via purposive sampling a heterogeneous population of users (N = 24) was recruited. Eye tracking was used to measure gaze behavior.

RESULTS

Overall, participants were satisfied with the tool (scale 0-10, 7.5; SD = 1.29). User friendliness of the mobile version (68.3; SD = 21.6) was higher than the web version (50.9; SD = 17.3) measured by SUS score (0-100). With regard to accessibility, eye tracking scenarios showed that the menu bar was hard to find (17% mobile, 55% web). In all scenario's, information was found faster in the mobile version than the web version. Attention was easily drawn to images. Regarding inclusiveness of the tool, we found significantly longer completing time of the scenario tasks for low language proficiency (p-value = 0.029) and higher age subgroups (p-value = 0.049). Lower language proficiency scored a significant lower SUS score (p-value = 0.012).

CONCLUSIONS

Overall, user friendliness and accessibility were positively evaluated. Assessment of inclusiveness emphasized the need for tailoring digital tools to those with low language proficiency and/or an older age. Co-creation of digital care tools with users is therefore important to match users' needs, make tools easily understandable and accessible to all users, and ultimately result in better uptake and impact.

Global Scientific Research Landscape on Medical Informatics From 2011 to 2020: Bibliometric Analysis

Background

With the emerging information and communication technology, the field of medical informatics has dramatically evolved in health care and medicine. Thus, it is crucial to explore the global scientific research landscape on medical informatics.

Objective

This study aims to present a visual form to clarify the overall scientific research trends of medical informatics in the past decade.

Methods

A bibliometric analysis of data retrieved and extracted from the Web of Science Core Collection (WoSCC) database was performed to analyze global scientific research trends on medical informatics, including publication year, journals, authors, institutions, countries/regions, references, and keywords, from January 1, 2011, to December 31, 2020.

Results

The data set recorded 34,742 articles related to medical informatics from WoSCC between 2011 and 2020. The annual global publications increased by 193.86% from 1987 in 2011 to 5839 in 2020. Journal of Medical Internet Research (3600 publications and 63,932 citations) was the most productive and most highly cited journal in the field of medical informatics. David W Bates (99 publications), Harvard University (1161 publications), and the United States (12,927 publications) were the most productive author, institution, and country, respectively. The co-occurrence cluster analysis of high-frequency author keywords formed 4 clusters: (1) artificial intelligence in health care and medicine; (2) mobile health; (3) implementation and evaluation of electronic health records; (4) medical informatics technology application in public health. COVID-19, which ranked third in 2020, was the emerging theme of medical informatics.

Conclusions

We summarize the recent advances in medical informatics in the past decade and shed light on their publication trends, influential journals, global collaboration patterns, basic knowledge, research hotspots, and theme evolution through bibliometric analysis and visualization maps. These findings will accurately and quickly grasp the research trends and provide valuable guidance for future medical informatics research.

Usability Testing of Virtual Reality Applications-The Pilot Study

The need for objective data-driven usability testing of VR applications is becoming more tangible with the rapid development of numerous VR applications and their increased accessibility. Traditional methods of testing are too time and resource consuming and might provide results that are highly subjective. Thus, the aim of this article is to explore the possibility of automation of usability testing of VR applications by using objective features such as HMD built-in head and hands tracking, EEG sensor, video recording, and other measurable parameters in addition to automated analysis of subjective data provided in questionnaires. For this purpose, a simple VR application was created which comprised relatively easy tasks that did not generate stress for the users. Fourteen volunteers took part in the study and their signals were monitored to acquire objective automated data. At the same time the observer was taking notes of subjects’ behaviour, and their subjective opinions about the experience were recorded in a post-experiment questionnaire. The results acquired from signal monitoring and questionnaires were juxtaposed with observation and post-interview results to confirm the validity and efficacy of automated usability testing. The results were very promising, proving that automated usability testing of VR applications is potentially achievable.

Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems

BACKGROUND AND OBJECTIVES

Electronic healthcare records have become central to patient care. Evaluation of new systems include a variety of usability evaluation methods or usability metrics (often referred to interchangeably as usability components or usability attributes). This study reviews the breadth of usability evaluation methods, metrics, and associated measurement techniques that have been reported to assess systems designed for hospital staff to assess inpatient clinical condition.

METHODS

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology, we searched Medline, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, and Open Grey from 1986 to 2019. For included studies, we recorded usability evaluation methods or usability metrics as appropriate, and any measurement techniques applied to illustrate these. We classified and described all usability evaluation methods, usability metrics, and measurement techniques. Study quality was evaluated using a modified Downs and Black checklist.

RESULTS

The search identified 1336 studies. After abstract screening, 130 full texts were reviewed. In the 51 included studies 11 distinct usability evaluation methods were identified. Within these usability evaluation methods, seven usability metrics were reported. The most common metrics were ISO9241-11 and Nielsen's components. An additional "usefulness" metric was reported in almost 40% of included studies. We identified 70 measurement techniques used to evaluate systems. Overall study quality was reflected in a mean modified Downs and Black checklist score of 6.8/10 (range 1-9) 33% studies classified as "high-quality" (scoring eight or higher), 51% studies "moderate-quality" (scoring 6-7), and the remaining 16% (scoring below five) were "low-quality."

CONCLUSION

There is little consistency within the field of electronic health record systems evaluation. This review highlights the variability within usability methods, metrics, and reporting. Standardized processes may improve evaluation and comparison electronic health record systems and improve their development and implementation.

Physical and information workflow mapping of vancomycin therapeutic drug management: A single site case study revealing potential gaps in the process

Vancomycin is one of the most prescribed antibiotics in pediatric intensive care units (PICU) in US hospitals. However, a detailed understanding of workflow and information flow among various stakeholders regarding vancomycin treatment processes in clinical settings is lacking. We conducted direct observations and informant interviews to develop the mapping of key processes and information flow for vancomycin treatment, with an emphasis on therapeutic drug monitoring (TDM) dose adjustment decision-making. A health information technology (HIT) sociotechnical framework was used to identify EHR related safety concerns. A total of 27 vancomycin treatment activities were observed over a 60-h duration including infusion administration, infusion completion, trough concentration blood draw and therapeutic decision making processes. Workflow and information flow mappings revealed (1) deviations between the documented timestamp used for TDM decision making and the actual time the tasks executed and (2) the lack of information flow regarding infusion completion and interruption. Missing features, insufficient usability and lack of integration with workflow and communication in the EHR were deemed safety gaps that may affect the accuracy of therapeutic decisions. Our case study identified gaps in information flow among clinical team members via EHR in TDM processes to provide insights for the improvement of the EHR system for antibiotic treatment purposes. In particular, the potential harm of the missing, uncertain, and inaccurate documented TDM task times warrant further investigations.

Investigation of usability problems of electronic medical record systems in the emergency department

BACKGROUND

Despite the benefits of using electronic medical record (EMR) systems, existing studies show that many healthcare providers are uncertain regarding their usability. The usability issues of these systems decrease their efficiency, discourage clinicians, and cause dissatisfaction among patients, which may result in safety risks and harm.

OBJECTIVE

The aim of this study was to collect and analyze EMR system usability problems from actual users. Practical user interface guidelines were presented based on the medical practices of these users.

METHODS

Employing an online questionnaire with a seven-point Likert scale, usability issues of EMR systems were collected from 200 emergency department healthcare providers (103 physicians (medical doctors) and 97 nurses) from South Korea.

RESULTS

The most common usability problem among the physicians and nurses was generating in-patient selection. This pertained to the difficulty in finding the required information on-screen because of poor visibility and a lack of distinctiveness.

CONCLUSIONS

The major problems of EMR systems and their causes were identified. It is recommended that intensive visual enhancement of EMR system interfaces should be implemented to support user tasks. By providing a better understanding of the current usability problems among medical practitioners, the results of this study can be useful for developing EMR systems with increased effectiveness and efficiency.

Too much or too little? Investigating the usability of high and low data displays of the same electronic medical record

The high data density on electronic medical record screens is touted as a major usability issue. However, it may not be a problem if the data is relevant and well-organized. Our objective was to test this assumption using a comprehensive set of measures that assess the three pillars of usability: efficiency (both physical and cognitive), effectiveness, and satisfaction. Physicians were asked to go through a series of tasks using two versions of the same electronic medical record: one where all the display items were separated into tabs (the original display), and one where important display items were grouped logically in one tab (the redesigned display). Results supported the hypothesis that combining relevant data in organized fashion into a smaller location would improve usability. The findings highlight the role of good display organization to mitigate the effects of high data density, as well as the importance of assessing cognitive load as part of usability studies.

Micro ad-hoc Health Social Networks (uHSN). Design and evaluation of a social-based solution for patient support

OBJECTIVE

To contribute the design, development, and assessment of a new concept: Micro ad hoc Health Social Networks (uHSN), to create a social-based solution for supporting patients with chronic disease.

DESIGN

After in-depth fieldwork and intensive co-design over a 4-year project following Community-Based Participatory Research (CBPR), this paper contributes a new paradigm of uHSN, defining two interaction areas (the "backstage", the sphere invisible to the final user, where processes that build services take place; and the "onstage", the visible part that includes the patients and relatives), and describes a new transversal concept, i.e., "network spaces segments," to provide timely interaction among all involved profiles and guaranteeing qualitative relationships. This proposal is applicable to any service design project and to all types of work areas; in the present work, it served as a social-based solution for supporting patients with chronic disease in two real-life health scenarios: a Parkinson disease patient association and a Stroke rehabilitation service in a hospital. These two scenarios included the following main features: thematic (related to the specific disease), private, and secure (only for the patient, relatives, healthcare professional, therapist, carer), with defined specific objectives (around patient support), small size (from tens to hundreds of users), ability to integrate innovative services (e.g., connection to hospital information service or to health sensors), supported by local therapeutic associations, and clustered with preconfigured relationships among users based in network groups.

MEASUREMENTS

Using a mixed qualitative and quantitative approach for 6 months, the performance of the uHSN was assessed in the two environments: a hospital rehabilitation unit working with Stroke patients, and a Parkinson disease association providing physiotherapy, occupational therapy, psychological support, speech therapy, and social services. We describe the proposed methods for evaluating the uHSN quantitatively and qualitatively, and how the scientific community can replicate and/or integrate this contribution in its research.

RESULTS

The uHSN overcomes the main limitations of traditional HSNs in the main areas recommended in the literature: privacy, security, transparency, system ecology, Quality of Service (QoS), and technology enhancement. The qualitative and quantitative research demonstrated its viability and replicability in four key points: user acceptance, productivity improvement, QoS enhancement, and fostering of social relations. It also meets the expectation of connecting health and social worlds, supporting distance rehabilitation, improving professionals' efficiency, expanding users' social capital, improving information quality and immediacy, and enhancing perceived peer/social/emotional support. The scientific contributions of the present paper are the first step not only in customizing health solutions that empower patients, their families, and healthcare professionals, but also in transferring this new paradigm to other scientific, professional, and social environments to create new opportunities.

Bridging the Gap Between Academic Research and Pragmatic Needs in Usability: A Hybrid Approach to Usability Evaluation of Health Care Information Systems

BACKGROUND

Technology is increasingly embedded into the full spectrum of health care. This movement has benefited from the application of software development practices such as usability testing and agile development processes. These practices are frequently applied in both commercial or operational and academic settings. However, the relative importance placed on rapid iteration, validity, reproducibility, generalizability, and efficiency differs between the 2 settings and the needs and objectives of academic versus pragmatic usability evaluations.

OBJECTIVE

This paper explores how usability evaluation typically varies on key dimensions in pragmatic versus academic settings that impact the rapidity, validity, and reproducibility of findings and proposes a hybrid approach aimed at satisfying both pragmatic and academic objectives.

METHODS

We outline the characteristics of pragmatic versus academically oriented usability testing in health care, describe the tensions and gaps resulting from differing contexts and goals, and present a model of this hybrid process along with 2 case studies of digital development projects in which we demonstrate this integrated approach to usability evaluation.

RESULTS

The case studies presented illustrate design choices characteristic of our hybrid approach to usability evaluation.

CONCLUSIONS

Designed to leverage the strengths of both pragmatically and academically focused usability studies, a hybrid approach allows new development projects to efficiently iterate and optimize from usability data as well as preserves the ability of these projects to produce deeper insights via thorough qualitative analysis to inform further tool development and usability research by way of academically focused dissemination.

Usability Evaluation of an Admission, Discharge, and Transfer Information System: A Heuristic Evaluation

BACKGROUND:

Admission, discharge and, transfer (ADT) process is one of the most important hospital workflows. ADT system is a part of a hospital information system (HIS).

AIM:

The objective of this study was to evaluate the usability of the ADT system.

METHODS:

The study performed at Mashhad University of Medical Sciences (MUMS) hospitals. Data collection instrument was a validated checklist of Pierotti heuristic evaluation. To determine the severity of usability problems, a hybrid of Nielson and Tampere unit for computer-human interaction (TAUCHI) severity scaling algorithm was used. Usability problems were divided into five categories (major, severe, minor, cosmetic, and technical). Six experts evaluated the ADT system independently. According to TAUCHI severity scale, if a feature has not yet been implemented in the ADT system, evaluators considered it a technical usability problem. Therefore, usability problems due to non-design feature in the ADT system were identified. Finally, the mean severity of each usability problems was calculated.

RESULTS:

A total of 186 usability problems were identified. The frequency of major, sever, minor and cosmetic usability problems were 2, 65, 69 and 50, respectively. A total of 55 usability problems by the evaluators were recognised as technical problems. The highest mismatch with usability principles was related to the “recognition rather than recall”. The range of the mean severity of usability problems was between 0-2.31.

CONCLUSIONS:

Our result showed that although implementation of IHIS on a large scale, it still suffered from unresolved usability problems. Identification of usability problems and evaluation of their level of severity, which was simultaneously performed in this study, can be used as a guide to evaluate the usability of other HISs.

A multi-level usability evaluation of mobile health applications: A case study

OBJECTIVE

To report a methodological approach for the development of a usable mHealth application (app).

MATERIALS AND METHODS

This work was guided by a 3-level stratified view of health information technology (IT) usability evaluation framework. We first describe a number of methodologies for operationalizing each level of the framework. Following the description of each methodology, we present a case study which illustrates the use of our preferred methodologies for the development of a mHealth app. At level 1 (user-task), we applied a card sorting technique to guide the information architecture of a mobile HIV symptom self-management app, entitled mVIP. At level 2 (user-task-system), we conducted a usability evaluation of mVIP in a laboratory setting through end-user usability testing and heuristic evaluation with informatics experts. At level 3 (user-task-system-environment), usability of mVIP was evaluated in a real-world setting following the use of the app during a 3-month trial.

RESULTS

The 3-level usability evaluation guided our work exploring in-depth interactions between the user, task, system, and environment. Integral to the findings from the 3-level usability evaluation, we iteratively refined the app's content, functionality, and interface to meet the needs of our intended end-users.

DISCUSSION AND CONCLUSION

The stratified view of the health IT usability evaluation framework is a useful methodological approach for the design, development, and evaluation of mHealth apps. The methodological recommendations for using the theoretical framework can inform future usability studies of mHealth apps.

Software for Administering the National Cancer Institute's Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events: Usability Study

Background

The US National Cancer Institute (NCI) developed software to gather symptomatic adverse events directly from patients participating in clinical trials. The software administers surveys to patients using items from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) through Web-based or automated telephone interfaces and facilitates the management of survey administration and the resultant data by professionals (clinicians and research associates).

Objective

The purpose of this study was to iteratively evaluate and improve the usability of the PRO-CTCAE software.

Methods

Heuristic evaluation of the software functionality was followed by semiscripted, think-aloud protocols in two consecutive rounds of usability testing among patients with cancer, clinicians, and research associates at 3 cancer centers. We conducted testing with patients both in clinics and at home (remotely) for both Web-based and telephone interfaces. Furthermore, we refined the software between rounds and retested.

Results

Heuristic evaluation identified deviations from the best practices across 10 standardized categories, which informed initial software improvement. Subsequently, we conducted user-based testing among 169 patients and 47 professionals. Software modifications between rounds addressed identified issues, including difficulty using radio buttons, absence of survey progress indicators, and login problems (for patients) as well as scheduling of patient surveys (for professionals). The initial System Usability Scale (SUS) score for the patient Web-based interface was 86 and 82 (P=.22) before and after modifications, respectively, whereas the task completion score was 4.47, which improved to 4.58 (P=.39) after modifications. Following modifications for professional users, the SUS scores improved from 71 to 75 (P=.47), and the mean task performance improved significantly (4.40 vs 4.02; P=.001).

Conclusions

Software modifications, informed by rigorous assessment, rendered a usable system, which is currently used in multiple NCI-sponsored multicenter cancer clinical trials.

Trial Registration

ClinicalTrials.gov NCT01031641; https://clinicaltrials.gov/ct2/show/NCT01031641 (Archived by WebCite at http://www.webcitation.org/708hTjlTl)

Comparison of two heuristic evaluation methods for evaluating the usability of health information systems

OBJECTIVES

In addition to following the usual Heuristic Evaluation (HE) method, the usability of health information systems can also be evaluated using a checklist. The objective of this study is to compare the performance of these two methods in identifying usability problems of health information systems.

METHODS

Eight evaluators independently evaluated different parts of a Medical Records Information System using two methods of HE (usual and with a checklist). The two methods were compared in terms of the number of problems identified, problem type, and the severity of identified problems.

RESULTS

In all, 192 usability problems were identified by two methods in the Medical Records Information System. This was significantly higher than the number of usability problems identified by the checklist and usual method (148 and 92, respectively) (p < 0.0001). After removing the duplicates, the difference between the number of unique usability problems identified by the checklist method (n = 100) and usual method (n = 44) was significant (p < 0.0001). Differences between the mean severity of the real usability problems (1.83) and those identified by only one of the methods (usual = 2.05, checklist = 1.74) were significant (p = 0.001).

CONCLUSIONS

This study revealed the potential of the two HE methods for identifying usability problems of health information systems. The results demonstrated that the checklist method had significantly better performance in terms of the number of identified usability problems; however, the performance of the usual method for identifying problems of higher severity was significantly better. Although the checklist method can be more efficient for less experienced evaluators, wherever usability is critical, the checklist should be used with caution in usability evaluations.

Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety

INTRODUCTION

Electronic audit and feedback (e-A&F) systems are used worldwide for care quality improvement. They measure health professionals' performance against clinical guidelines, and some systems suggest improvement actions. However, little is known about optimal interface designs for e-A&F, in particular how to present suggested actions for improvement. We developed a novel theory-informed system for primary care (the Performance Improvement plaN GeneratoR; PINGR) that covers the four principal interface components: clinical performance summaries; patient lists; detailed patient-level information; and suggested actions. As far as we are aware, this is the first report of an e-A&F system with all four interface components.

OBJECTIVES

(1) Use a combination of quantitative and qualitative methods to evaluate the usability of PINGR with target end-users; (2) refine existing design recommendations for e-A&F systems; (3) determine the implications of these recommendations for patient safety.

METHODS

We recruited seven primary care physicians to perform seven tasks with PINGR, during which we measured on-screen behaviour and eye movements. Participants subsequently completed usability questionnaires, and were interviewed in-depth. Data were integrated to: gain a more complete understanding of usability issues; enhance and explain each other's findings; and triangulate results to increase validity.

RESULTS

Participants committed a median of 10 errors (range 8-21) when using PINGR's interface, and completed a median of five out of seven tasks (range 4-7). Errors violated six usability heuristics: clear response options; perceptual grouping and data relationships; representational formats; unambiguous description; visually distinct screens for confusable items; and workflow integration. Eye movement analysis revealed the integration of components largely supported effective user workflow, although the modular design of clinical performance summaries unnecessarily increased cognitive load. Interviews and questionnaires revealed PINGR is user-friendly, and that improved information prioritisation could further promote useful user action.

CONCLUSIONS

Comparing our results with the wider usability literature we refine a previously published set of interface design recommendations for e-A&F. The implications for patient safety are significant regarding: user engagement; actionability; and information prioritisation. Our results also support adopting multi-method approaches in usability studies to maximise issue discovery and the credibility of findings.

Users’ design feedback in usability evaluation: a literature review

As part of usability evaluation, users may be invited to offer their reflections on the system being evaluated. Such reflections may concern the system’s suitability for its context of use, usability problem predictions, and design suggestions. We term the data resulting from such reflections users’ design feedback. Gathering users’ design feedback as part of usability evaluation may be seen as controversial, and the current knowledge on users’ design feedback is fragmented. To mitigate this, we have conducted a literature review. The review provides an overview of the benefits and limitations of users’ design feedback in usability evaluations. Following an extensive search process, 31 research papers were identified as relevant and analysed. Users’ design feedback is gathered for a number of distinct purposes: to support budget approaches to usability testing, to expand on interaction data from usability testing, to provide insight into usability problems in users’ everyday context, and to benefit from users’ knowledge and creativity. Evaluation findings based on users’ design feedback can be qualitatively different from, and hence complement, findings based on other types of evaluation data. Furthermore, findings based on users’ design feedback can hold acceptable validity, though the thoroughness of such findings may be questioned. Finally, findings from users’ design feedback may have substantial impact in the downstream development process. Four practical implications are highlighted, and three directions for future research are suggested.

Iterative User Interface Design for Automated Sequential Organ Failure Assessment Score Calculator in Sepsis Detection

Background

The new sepsis definition has increased the need for frequent sequential organ failure assessment (SOFA) score recalculation and the clerical burden of information retrieval makes this score ideal for automated calculation.

Objective

The aim of this study was to (1) estimate the clerical workload of manual SOFA score calculation through a time-motion analysis and (2) describe a user-centered design process for an electronic medical record (EMR) integrated, automated SOFA score calculator with subsequent usability evaluation study.

Methods

First, we performed a time-motion analysis by recording time-to-task-completion for the manual calculation of 35 baseline and 35 current SOFA scores by 14 internal medicine residents over a 2-month period. Next, we used an agile development process to create a user interface for a previously developed automated SOFA score calculator. The final user interface usability was evaluated by clinician end users with the Computer Systems Usability Questionnaire.

Results

The overall mean (standard deviation, SD) time-to-complete manual SOFA score calculation time was 61.6 s (33). Among the 24% (12/50) usability survey respondents, our user-centered user interface design process resulted in >75% favorability of survey items in the domains of system usability, information quality, and interface quality.

Conclusions

Early stakeholder engagement in our agile design process resulted in a user interface for an automated SOFA score calculator that reduced clinician workload and met clinicians’ needs at the point of care. Emerging interoperable platforms may facilitate dissemination of similarly useful clinical score calculators and decision support algorithms as “apps.” A user-centered design process and usability evaluation should be considered during creation of these tools.

An Iterative, Mixed Usability Approach Applied to the Telekit System from the Danish TeleCare North Trial

Objective. The aim of the present study is to evaluate the usability of the telehealth system, coined Telekit, by using an iterative, mixed usability approach. Materials and Methods. Ten double experts participated in two heuristic evaluations (HE1, HE2), and 11 COPD patients attended two think-aloud tests. The double experts identified usability violations and classified them into Jakob Nielsen's heuristics. These violations were then translated into measurable values on a scale of 0 to 4 indicating degree of severity. In the think-aloud tests, COPD participants were invited to verbalise their thoughts. Results. The double experts identified 86 usability violations in HE1 and 101 usability violations in HE2. The majority of the violations were rated in the 0–2 range. The findings from the think-aloud tests resulted in 12 themes and associated examples regarding the usability of the Telekit system. The use of the iterative, mixed usability approach produced both quantitative and qualitative results. Conclusion. The iterative, mixed usability approach yields a strong result owing to the high number of problems identified in the tests because the double experts and the COPD participants focus on different aspects of Telekit's usability. This trial is registered with Clinicaltrials.gov, NCT01984840, November 14, 2013.

The EU-project United4Health: User-centred design of an information system for a Norwegian telemedicine service

Introduction

Organizational changes of health care services in Norway brought to light a need for new clinical pathways. This study presents the design and evaluation of an information system for a new telemedicine service for chronic obstructive pulmonary disease patients after hospital discharge.

Methods

A user-centred design approach was employed composed of a workshop with end-users, two user tests and a field trial. For data collection, qualitative methods such as observations, semi-structured interviews and a questionnaire were used.

Results

User workshop’s outcome informed the implementation of the system initial prototype, evaluated by end-users in a usability laboratory. Several usability and functionality issues were identified and solved, such as the interface between the initial colour scheme and the triage colours. Iterative refinements were made and a second user evaluation showed that the main issues were solved. The responses to a questionnaire presented a high score of user satisfaction. In the final phase, a field trial showed satisfactory use of the system.

Discussion

This study showed how the target end-users groups were actively involved in identifying the needs, suggestions and preferences. These aspects were addressed in the development of an information system through a user-centred design process. The process efficiently enabled users to give feedback about design and functionality. Continuous refinement of the system was the key to full development and suitability for the telemedicine service. This research was a result of the international cooperation between partners within the project United4Health, a part of the Seventh Framework Programme for Research of the European Union.

An appraisal of published usability evaluations of electronic health records via systematic review

OBJECTIVE

In this systematic review, we aimed to evaluate methodological and reporting trends present in the current literature by investigating published usability studies of electronic health records (EHRs).

METHODS

A literature search was conducted for articles published through January 2015 using MEDLINE (Ovid), EMBASE, Scopus, and Web of Science, supplemented by citation and reference list reviews. Studies were included if they tested the usability of hospital and clinic EHR systems in the inpatient, outpatient, emergency department, or operating room setting.

RESULTS

A total of 4848 references were identified for title and abstract screening. Full text screening was performed for 197 articles, with 120 meeting the criteria for study inclusion.

CONCLUSION

A review of the literature demonstrates a paucity of quality published studies describing scientifically valid and reproducible usability evaluations at various stages of EHR system development. A lack of formal and standardized reporting of EHR usability evaluation results is a major contributor to this knowledge gap, and efforts to improve this deficiency will be one step of moving the field of usability engineering forward.

An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach

OBJECTIVE

mHealth systems are becoming more common to aid patients in their diabetes self-management, but recent studies indicate a need for thorough evaluation of patients' experienced usability. Current evaluations lack a multi-method design for data collection and structured methods for data analyses. The purpose of this study was to provide a feasibility test of a multi-method approach for both data collection and data analyses for patients' experienced usability of a mHealth system for diabetes type 2 self-management.

MATERIALS AND METHODS

A random sample of 10 users was selected from a larger clinical trial. Data collection methods included user testing with eight representative tasks and Think Aloud protocol, a semi-structured interview and a questionnaire on patients' experiences using the system. The Framework Analysis (FA) method and Usability Problem Taxonomy (UPT) were used to structure, code and analyze the results. A usability severity rating was assigned after classification.

RESULTS

The combined methods resulted in a total of 117 problems condensed into 19 usability issues with an average severity rating of 2.47 or serious. The usability test detected 50% of the initial usability problems, followed by the post-interview at 29%. The usability test found 18 of 19 consolidated usability problems while the questionnaire uncovered one unique issue. Patients experienced most usability problems (8) in the Glucose Readings View when performing complex tasks such as adding, deleting, and exporting glucose measurements. The severity ratings were the highest for the Glucose Diary View, Glucose Readings View, and Blood Pressure View with an average severity rating of 3 (serious). Most of the issues were classified under the artifact component of the UPT and primary categories of Visualness (7) and Manipulation (6). In the UPT task component, most issues were in the primary category Task-mapping (12).

CONCLUSIONS

Multiple data collection methods yielded a more comprehensive set of usability issues. Usability testing uncovered the largest volume of usability issues, followed by interviewing and then the questionnaire. The interview did not surface any unique consolidated usability issues while the questionnaire surfaced one. The FA and UPT were valuable in structuring and classifying problems. The resulting descriptions serve as a communication tool in problem solving and programming. We recommend the usage of multiple methods in data collection and employing the FA and UPT in data analyses for future usability testing.

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework

Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of "the best available" comparison system to enable a cognitive walkthrough in phase 1 and a comparative effectiveness evaluation in phase 2. During phase 2, five quantitative and qualitative evaluation methods are mixed to assess usability: time-motion analysis, software log, questionnaires - System Usability Scale and the Unified Theory of Acceptance of Use of Technology, think-aloud protocols, and unstructured interviews. Each method contributes data for a unique measure (e.g., time motion analysis contributes task-completion-time; software log contributes action transition frequency). The measures are triangulated to yield complementary insights regarding user-perceived ease-of-use, functionality integration, anxiety during use, and workflow impact. To illustrate its use, we applied this framework in a formative evaluation of a software called Integrated Model for Patient Care and Clinical Trials (IMPACT). We conclude that this mixed-methods evaluation framework enables an integrated assessment of user needs satisfaction and user-perceived usefulness and usability of a novel design. This evaluation framework effectively bridges the gap between co-evolving user needs and technology designs during iterative prototyping and is particularly useful when it is difficult for users to articulate their needs for technology support due to the lack of a baseline.

Measures of user experience in a streptococcal pharyngitis and pneumonia clinical decision support tools

OBJECTIVE

To understand clinician adoption of CDS tools as this may provide important insights for the implementation and dissemination of future CDS tools.

MATERIALS AND METHODS

Clinicians (n=168) at a large academic center were randomized into intervention and control arms to assess the impact of strep and pneumonia CDS tools. Intervention arm data were analyzed to examine provider adoption and clinical workflow. Electronic health record data were collected on trigger location, the use of each component and whether an antibiotic, other medication or test was ordered. Frequencies were tabulated and regression analyses were used to determine the association of tool component use and physician orders.

RESULTS

The CDS tool was triggered 586 times over the study period. Diagnosis was the most frequent workflow trigger of the CDS tool (57%) as compared to chief complaint (30%) and diagnosis/antibiotic combinations (13%). Conversely, chief complaint was associated with the highest rate (83%) of triggers leading to an initiation of the CDS tool (opening the risk prediction calculator). Similar patterns were noted for initiation of the CDS bundled ordered set and completion of the entire CDS tool pathway. Completion of risk prediction and bundled order set components were associated with lower rates of antibiotic prescribing (OR 0.5; CI 0.2-1.2 and OR 0.5; CI 0.3-0.9, respectively).

DISCUSSION

Different CDS trigger points in the clinician user workflow lead to substantial variation in downstream use of the CDS tool components. These variations were important as they were associated with significant differences in antibiotic ordering.

CONCLUSIONS

These results highlight the importance of workflow integration and flexibility for CDS success.

The impacts of second generation e-prescribing usability on community pharmacists outcomes

BACKGROUND

The results from past studies about the effects of second-generation e-prescribing systems on community pharmacists' outcomes and practices are inconclusive, and the claims of effectiveness and efficiency of such systems have not been supported in all studies. There is a strong need to study the factors that lead to positive outcomes for the users of these systems.

OBJECTIVE

This paper intends to bridge the above gaps by empirically examining the impacts of user interface usability on the community pharmacists' outcomes.

METHODS

A quantitative survey research method was used and the data was collected from the community pharmacists, who use an e-prescribing system. Data from 152 questionnaires collected in a national survey were used to for the study. Partial Least Squares (PLS) path modeling was used to examine scale reliability, validity and hypotheses.

RESULTS

The scale was found to test well for reliability and validity. Examining the hypotheses illustrated that ease of use (P < 0.01, t = 5.79) and information quality (P < 0.01, t = 6.24) of an e-prescribing system improved pharmacists' outcomes (including communication, facilitation of care, reduction of workload and medical errors) while ease of use of the system was influenced by user interface consistency (P < 0.01, t = 7.35) and system error prevention (P < 0.01, t = 5.29).

CONCLUSION

To improve community pharmacists' outcomes and practices, the ease of use, information quality, consistency and error prevention features of e-prescribing systems should be improved. It was found that information quality had a stronger impact on the outcomes and hence improving the quality of the generated information would have higher impacts on users' outcomes.

Usability testing of Avoiding Diabetes Thru Action Plan Targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record

PURPOSE

Usability testing can be used to evaluate human-computer interaction (HCI) and communication in shared decision making (SDM) for patient-provider behavioral change and behavioral contracting. Traditional evaluations of usability using scripted or mock patient scenarios with think-aloud protocol analysis provide a way to identify HCI issues. In this paper we describe the application of these methods in the evaluation of the Avoiding Diabetes Thru Action Plan Targeting (ADAPT) tool, and test the usability of the tool to support the ADAPT framework for integrated care counseling of pre-diabetes. The think-aloud protocol analysis typically does not provide an assessment of how patient-provider interactions are effected in "live" clinical workflow or whether a tool is successful. Therefore, "Near-live" clinical simulations involving applied simulation methods were used to compliment the think-aloud results. This complementary usability technique was used to test the end-user HCI and tool performance by more closely mimicking the clinical workflow and capturing interaction sequences along with assessing the functionality of computer module prototypes on clinician workflow. We expected this method to further complement and provide different usability findings as compared to think-aloud analysis. Together, this mixed method evaluation provided comprehensive and realistic feedback for iterative refinement of the ADAPT system prior to implementation.

METHODS

The study employed two phases of testing of a new interactive ADAPT tool that embedded an evidence-based shared goal setting component into primary care workflow for dealing with pre-diabetes counseling within a commercial physician office electronic health record (EHR). Phase I applied usability testing that involved "think-aloud" protocol analysis of eight primary care providers interacting with several scripted clinical scenarios. Phase II used "near-live" clinical simulations of five providers interacting with standardized trained patient actors enacting the clinical scenario of counseling for pre-diabetes, each of whom had a pedometer that recorded the number of steps taken over a week. In both phases, all sessions were audio-taped and motion screen-capture software was activated for onscreen recordings. Transcripts were coded using iterative qualitative content analysis methods.

RESULTS

In Phase I, the impact of the components and layout of ADAPT on user's Navigation, Understandability, and Workflow were associated with the largest volume of negative comments (i.e. approximately 80% of end-user commentary), while Usability and Content of ADAPT were representative of more positive than negative user commentary. The heuristic category of Usability had a positive-to-negative comment ratio of 2.1, reflecting positive perception of the usability of the tool, its functionality, and overall co-productive utilization of ADAPT. However, there were mixed perceptions about content (i.e., how the information was displayed, organized and described in the tool). In Phase II, the duration of patient encounters was approximately 10 min with all of the Patient Instructions (prescriptions) and behavioral contracting being activated at the end of each visit. Upon activation, providers accepted the pathway prescribed by the tool 100% of the time and completed all the fields in the tool in the simulation cases. Only 14% of encounter time was spent using the functionality of the ADAPT tool in terms of keystrokes and entering relevant data. The rest of the time was spent on communication and dialog to populate the patient instructions. In all cases, the interaction sequence of reviewing and discussing exercise and diet of the patient was linked to the functionality of the ADAPT tool in terms of monitoring, response-efficacy, self-efficacy, and negotiation in the patient-provider dialog. There was a change from one-way dialog to two-way dialog and negotiation that ended in a behavioral contract. This change demonstrated the tool's sequence, which supported recording current exercise and diet followed by a diet and exercise goal setting procedure to reduce the risk of diabetes onset.

CONCLUSIONS

This study demonstrated that "think-aloud" protocol analysis with "near-live" clinical simulations provided a successful usability evaluation of a new primary care pre-diabetes shared goal setting tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to show the influence of the ADAPT framework on the usability, workflow integration, and communication between the patient and provider. The think-aloud tests with the provider showed the tool can be used according to the ADAPT framework (exercise-to-diet behavior change and tool utilization), while the clinical simulations revealed the ADAPT framework to realistically support patient-provider communication to obtain behavioral change contract. SDM interactions and mechanisms affecting protocol-based care can be more completely captured by combining "near-live" clinical simulations with traditional "think-aloud analysis" which augments clinician utilization. More analysis is required to verify if the rich communication actions found in Phase II compliment clinical workflows.

Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

OBJECTIVE

To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys.

MATERIALS AND METHODS

Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35).

RESULTS

The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods.

DISCUSSION

Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR.

CONCLUSION

We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

An observation tool for studying patient-oriented workflow in hospital emergency departments

BACKGROUND

Studying workflow is a critical step in designing, implementing and evaluating informatics interventions in complex sociotechnical settings, such as hospital emergency departments (EDs). Known approaches to studying workflow in clinical settings attend to the activities of individual clinicians, thus being inadequate to characterize patient care as a cooperative work.

OBJECTIVES

The purpose of this paper is twofold. First, we introduce a novel, theory-driven patient-oriented workflow methodology, which better addresses the complex, multiple-provider nature of patient care. Second, we report the development of an observational tool and protocol for use in studies of this type, and the results of an evaluation study.

METHODS

We created a tablet computer implementation of an instrument to efficiently capture patient-oriented workflow, and evaluated it through a field study in three EDs. We focused on activities occurring over time during a single patient care episode as well as the roles of the ED staff members who conducted the activities.

RESULTS

The evidence generated supports the validity, viability, and reliability of the tool. The coverage of the tool in terms of activities and roles was satisfactory. The tool was able to capture the sequence of activity-role pairs for 108 patient care episodes. The inter-rater reliability assessment yielded a high kappa value (0.79).

DISCUSSION

The patient-oriented workflow methodology has the potential to facilitate modeling patient care in EDs by characterizing both roles and activities in sequence. The methodology also provides researchers and practitioners a more realistic and comprehensive workflow perspective that can inform the design, implementation and evaluation of health information technology interventions.

Collection and visualization of dietary behavior and reasons for eating using Twitter

BACKGROUND

Increasing an individual's awareness and understanding of their dietary habits and reasons for eating may help facilitate positive dietary changes. Mobile technologies allow individuals to record diet-related behavior in real time from any location; however, the most popular software applications lack empirical evidence supporting their efficacy as health promotion tools.

OBJECTIVE

The purpose of this study was to test the feasibility and acceptability of a popular social media software application (Twitter) to capture young adults' dietary behavior and reasons for eating. A secondary aim was to visualize data from Twitter using a novel analytic tool designed to help identify relationships among dietary behaviors, reasons for eating, and contextual factors.

METHODS

Participants were trained to record all food and beverages consumed over 3 consecutive days (2 weekdays and 1 weekend day) using their mobile device's native Twitter application. A list of 24 hashtags (#) representing food groups and reasons for eating were provided to participants to guide reporting (eg, #protein, #mood). Participants were encouraged to annotate hashtags with contextual information using photos, text, and links. User experience was assessed through a combination of email reports of technical challenges and a 9-item exit survey. Participant data were captured from the public Twitter stream, and frequency of hashtag occurrence and co-occurrence were determined. Contextual data were further parsed and qualitatively analyzed. A frequency matrix was constructed to identify food and behavior hashtags that co-occurred. These relationships were visualized using GMap algorithmic mapping software.

RESULTS

A total of 50 adults completed the study. In all, 773 tweets including 2862 hashtags (1756 foods and 1106 reasons for eating) were reported. Frequently reported food groups were #grains (n=365 tweets), #dairy (n=221), and #protein (n=307). The most frequently cited reasons for eating were #social (activity) (n=122), #taste (n=146), and #convenience (n=173). Participants used a combination of study-provided hash tags and their own hash tags to describe behavior. Most rated Twitter as easy to use for the purpose of reporting diet-related behavior. "Maps" of hash tag occurrences and co-occurrences were developed that suggested time-varying diet and behavior patterns.

CONCLUSIONS

Twitter combined with an analytical software tool provides a method for capturing real-time food consumption and diet-related behavior. Data visualization may provide a method to identify relationships between dietary and behavioral factors. These findings will inform the design of a study exploring the use of social media and data visualization to identify relationships between food consumption, reasons for engaging in specific food-related behaviors, relevant contextual factors, and weight and health statuses in diverse populations.

Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA

In response to mounting evidence that use of electronic medical record systems may cause unintended consequences, and even patient harm, the AMIA Board of Directors convened a Task Force on Usability to examine evidence from the literature and make recommendations. This task force was composed of representatives from both academic settings and vendors of electronic health record (EHR) systems. After a careful review of the literature and of vendor experiences with EHR design and implementation, the task force developed 10 recommendations in four areas: (1) human factors health information technology (IT) research, (2) health IT policy, (3) industry recommendations, and (4) recommendations for the clinician end-user of EHR software. These AMIA recommendations are intended to stimulate informed debate, provide a plan to increase understanding of the impact of usability on the effective use of health IT, and lead to safer and higher quality care with the adoption of useful and usable EHR systems.

Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support

PURPOSE

Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow.

METHODS

This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods.

RESULTS

In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12 min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS.

CONCLUSIONS

This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

Quality of human-computer interaction--results of a national usability survey of hospital-IT in Germany

Background

Due to the increasing functionality of medical information systems, it is hard to imagine day to day work in hospitals without IT support. Therefore, the design of dialogues between humans and information systems is one of the most important issues to be addressed in health care. This survey presents an analysis of the current quality level of human-computer interaction of healthcare-IT in German hospitals, focused on the users' point of view.

Methods

To evaluate the usability of clinical-IT according to the design principles of EN ISO 9241-10 the IsoMetrics Inventory, an assessment tool, was used. The focus of this paper has been put on suitability for task, training effort and conformity with user expectations, differentiated by information systems. Effectiveness has been evaluated with the focus on interoperability and functionality of different IT systems.

Results

4521 persons from 371 hospitals visited the start page of the study, while 1003 persons from 158 hospitals completed the questionnaire. The results show relevant variations between different information systems.

Conclusions

Specialised information systems with defined functionality received better assessments than clinical information systems in general. This could be attributed to the improved customisation of these specialised systems for specific working environments. The results can be used as reference data for evaluation and benchmarking of human computer engineering in clinical health IT context for future studies.

National questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability

OBJECTIVE

In the health informatics field, usability studies typically focus on evaluating a single information system and involve a rather small group of end-users. However, little is known about the usability of clinical information and communication technology (ICT) environment in which healthcare professionals work daily. This paper aims at contributing to usability research and user-oriented development of healthcare technologies with three objectives: inform researchers and practitioners about the current state of usability of clinical ICT systems, increase the understanding of usability aspects specific for clinical context, and encourage a more holistic approach on studying usability issues in health informatics field.

METHODS

A national web questionnaire study was conducted in Finland in spring 2010 with 3929 physicians actively working in patient care. For the purposes of the study, we described three dimensions of clinical ICT system usability that reflect the physicians' viewpoint on system usage: (1) compatibility between clinical ICT systems and physicians' tasks, (2) ICT support for information exchange, communication and collaboration in clinical work, and (3) interoperability and reliability. The dimensions derive from the definitions of usability and clinical context of use analysis, and reflect the ability of ICT systems to have a positive impact on patient care by supporting physicians in achieving their goals with a pleasant user experience. The research data incorporated 32 statements with a five-point Likert-scale on physicians' experiences on usability of their currently used ICT systems and a summative question about school grade given to electronic health record (EHR) systems.

RESULTS

Physicians' estimates of their EHR systems were very critical. With the rating scale from 4 or fail to 10 or excellent, the average of the grades varied from 6.1 to 8.4 dependent on the kind of facility the physician is working. Questionnaire results indicated several usability problems and deficiencies which considerably hindered the efficiency of clinical ICT use and physician's routine work. Systems lacked the appropriate features to support typical clinical tasks, such as decision making, prevention of medical errors, and review of a patient's treatment chart. The systems also required physicians to perform fixed sequences of steps and tasks, and poorly supported the documentation and retrieval of patient data. The findings on ICT support for collaboration showed mainly negative results, aside from collaboration between co-located physicians. In addition, the study results pointed out physicians suffering from system failures and a lack of integration between the systems.

CONCLUSIONS

The described study and related results are unique in several ways. A national usability study with nearly 4000 respondents had not been conducted in other countries in which healthcare technologies are widely adopted. The questionnaire study provided a generalized picture about the usability problems, however, it should be noted that there were significant differences between legacy systems in use. Previously, researchers had not approached contextual aspects of usability the context of clinical work, where numerous systems are in use. The described usability dimensions and the presented study results can be considered as the first step towards conceptualizing ICT usability in the unique setting of clinical work.