Quality of human-computer interaction--results of a national usability survey of hospital-IT in Germany

Impact of response bias in three surveys on primary care providers' experiences with electronic health records

OBJECTIVE

To identify impacts of different survey methodologies assessing primary care physicians' (PCPs') experiences with electronic health records (EHRs), we compared three surveys: the 2022 Continuous Certification Questionnaire (CCQ) from the American Board of Family Medicine, the 2022 University of California San Francisco (UCSF) Physician Health IT Survey, and the 2021 National Electronic Health Records Survey (NEHRS).

MATERIALS AND METHODS

We evaluated differences between survey pairs using Rao-Scott corrected chi-square tests, which account for weighting.

RESULTS

CCQ received 3991 responses from PCPs (100% response rate), UCSF received 1375 (3.6% response rate), and NEHRS received 858 (18.2% response rate). Substantial, statistically significant differences in demographics were detected across the surveys. CCQ respondents were younger and more likely to work in a health system; NEHRS respondents were more likely to work in private practice; and UCSF respondents disproportionately practiced in larger academic settings. Many EHR experience indicators were similar between CCQ and NEHRS, but CCQ respondents reported higher documentation burden.

DISCUSSION

The UCSF approach is unlikely to supply reliable data. Significant demographic differences between CCQ and NEHRS raise response bias concerns, and while there were similarities in some reported EHR experiences, there were important, significant differences.

CONCLUSION

Federal EHR policy monitoring and maintenance require reliable data. This test of existing and alternative sources suggest that diversified data sources are necessary to understand physicians' experiences with EHRs and interoperability. Comprehensive surveys administered by specialty boards have the potential to contribute to these efforts, since they are likely to be free of response bias.

[Use of AI-based applications by hospital staff: task profiles and qualification requirements]

BACKGROUND

Artificial intelligence (AI) is becoming increasingly important for the future development of hospitals. To unlock the large potential of AI, job profiles of hospital staff members need to be further developed in the direction of AI and digitization skills through targeted qualification measures. This affects both medical and non-medical processes along the entire value chain in hospitals. The aim of this paper is to provide an overview of the skills required to deal with smart technologies in a clinical context and to present measures for training employees.

METHODS

As part of the "SmartHospital.NRW" project in 2022, we conducted a literature review as well as interviews and workshops with experts. AI technologies and fields of application were identified.

RESULTS

Key findings include adapted and new task profiles, synergies and dependencies between individual task profiles, and the need for a comprehensive interdisciplinary and interprofessional exchange when using AI-based applications in hospitals.

DISCUSSION

Our article shows that hospitals need to promote digital health literacy skills for hospital staff members at an early stage and at the same time recruit technology- and AI-savvy staff. Interprofessional exchange formats and accompanying change management are essential for the use of AI in hospitals.

National survey of user-reported usability of electronic medical record software in ophthalmology in Germany

PURPOSE

A nationwide, comparative survey of the physician-reported usability of electronic medical record (EMR) software used by ophthalmologists in Germany using the System Usability Scale (SUS) as a standardized metric.

METHODS

A cross-sectional survey of members of the German Ophthalmological Society (DOG) and professional association of ophthalmologists (BVA) was conducted in May 2022. All 7788 physician members of both societies were invited to participate in an anonymous online-survey by individualized links. User-reported usability of the participants main software used for electronic medical recordkeeping was assessed using the SUS (range 0-100).

RESULTS

A total of 881 participants with 51 different EMRs completed the entire questionnaire. Mean EMR-SUS score was 65.7 (SD ± 23.5). Significant differences in mean SUS of several EMR programs were observed with a range of 31.5 to 87.2 in programs with 10 or more responses. 31.8% of all main program SUS ratings were below 50 points. Female gender was associated with 4.02 higher SUS score (95% CI 0.46-7.59). Main program SUS was positively correlated with overall work-related satisfaction and work environment SUS but negatively correlated with the number of programs in the work environment. The SUS of the entire digital work environment including all programs used daily was closely correlated with the main EMR SUS, but not the number of programs used.

CONCLUSION

Our survey revealed a fragmented pattern of EMR use by ophthalmologists in Germany with many competing software products and widely diverging mean System Usability Scale scores. A considerable share of ophthalmologists report EMR usability below what is commonly considered acceptable.

A National Survey of EMR Usability: Comparisons between medical and nursing professions in the hospital and primary care sectors in Australia and Finland

BACKGROUND

Electronic Medical Record Systems (EMRs) are now part of nursing and medical professionals daily work in the acute and primary care sectors in Australia. Usability is an important factor in their successful adoption and impacts upon clinical workflow, safety and quality, communication, and collaboration. This study replicates a significant body of work conducted by Finnish researchers applying a usability focused survey to understand medical and nursing professionals' experiences in the Australian context. As we implement EMRs across health systems, their usability and design to support clinicians to effectively deliver and document care, is essential.

METHODS

We conducted an observational study using a cross sectional survey, the National Usability-Focused HIS Scale (NuHISS) developed and validated by Finnish researchers. For this study 13 usability statements collected clinician impressions of EMRs related to technical quality, ease of use, benefits, and collaboration. We report the responses from medical and nursing professionals working in clinical practice settings in Australia, including primary care and hospital sectors in 2020.

RESULTS

Nursing and medical professionals have different experiences with EMR usability. This depends on the sector they work in and the usability feature measured. In our sample, technical quality features were more positively experienced by doctors in the primary care sector than nurses as well as ease of obtaining patient information and prevention of errors. In the hospital sector nurses experiences with EMRs were more positive with respect to support for routine task completion, learnability, ease of obtaining patient information and entry of patient data.

CONCLUSIONS

The NuHISS is a suitable tool for measuring the usability experiences of Australian clinicians and the EMRs utilised. Differences in usability experiences were noted between professional groups and sectors. A focus on the usability perspectives of clinicians when enhancing or developing EMR solutions is advocated.

Why Are Digital Health Care Systems Still Poorly Designed, and Why Is Health Care Practice Not Asking for More? Three Paths Toward a Sustainable Digital Work Environment

Knowledge of how to design digital systems that are ergonomically sound, high in usability, and optimized for the user, context, and task has existed for some time. Despite this, there are still too many examples of new digital health care systems that are poorly designed and that could negatively affect both the work environment of health care staff and patient safety. This could be because of a gap between the theoretical knowledge of design and ergonomics and the practical implementation of this knowledge in procuring and developing digital health care systems. Furthermore, discussions of digitalization are often at a general level and risk neglecting the nature of direct interaction with the digital system. This is problematic since it is at this detailed level that work environment and patient safety issues materialize in practice. In this paper, we illustrate such issues with two scenarios concerned with contemporary electronic health care records, based on field studies in two health care settings. We argue that current methods and tools for designing and evaluating digital systems in health care must cater both to the holistic level and to the details of interaction and ergonomics. It must also be acknowledged that health care professionals are neither designers nor engineers, so expectations of them during the development of digital systems must be realistic. We suggest three paths toward a more sustainable digital work environment in health care: (1) better tools for evaluating the digital work environment in the field; (2) generic formulations of qualitative requirements related to usability and for adaptation to the user, context, and task, to be used in procurement; and (3) the introduction of digital ergonomics as an embracing concept capturing several of the ergonomic challenges (including physical, cognitive, and organizational aspects) involved in implementing and using digital systems.

Electronic Health Records in Specialized Pediatric Palliative Care: A Qualitative Needs Assessment among Professionals Experienced and Inexperienced in Electronic Documentation

BACKGROUND

Currently, to the best of our knowledge, no findings exist concerning the needs of professionals in specialized pediatric palliative care (PPC) regarding electronic health records (EHRs). Several studies have highlighted benefits concerning the use of EHRs in pediatrics. However, usability is strongly affected by the degree of adaptivity to the context of application. The aim of this study is to examine the needs of professionals concerning an EHR in the specialized PPC inpatient and outpatient settings.

METHODS

A qualitative research design was chosen to address the complex aspects of user demands. Focus group interviews and semi-structured one-on-one interviews were conducted with PPC professionals. N = 23 participants from inpatient and N = 11 participants from outpatient settings of specialized PPC representing various professions took part in the study.

RESULTS

The findings could be grouped into four categories: (1) attitude towards the current methods of documentation, (2) attitude towards electronic documentation in general, (3) general requirements for an EHR, and (4) content requirements for an EHR.

CONCLUSIONS

Professionals in specialized PPC expect and experience many benefits of using electronic documentation. Their requirements for an EHR for inpatient and outpatient settings of PPC are largely consistent with EHRs for pediatrics. However, individual specifications and adaptations are necessary for this particular setting.

Usability of electronic patient record systems: Instrument validation study conducted for hospitals in Germany

In Germany and elsewhere, few hospital electronic patient record usability surveys are available. Moreover, there seems to be a complete lack of validated instruments in the respective literature. Hence, this study’s purpose is to validate a scale for measuring the usability of hospital electronic patient record systems. The data used for the study’s analyses (n = 949) were originally obtained during the German national initiative ‘Hospital IT User Questionnaire’. In the course of the study, reliability and exploratory factor analyses were conducted and psychometric tests showed a reliable, valid and suitable instrument. Descriptive data analysis suggests a generally low user perception and variances hint at a high potential for future improvements. To our knowledge, this study shows that a general inventory (subscales from the IsoMetrics inventory) can be applied to measuring hospital electronic patient record usability as well. The validated instrument can be used to inform healthcare providers, decision makers and politicians of a given state of usability, discrepancies between different hospitals or systems providers, and serve as a basis for improvements.

[Comparison of current critical care information systems from the perspective of clinical users : Summary of the results of a German nationwide survey]

BACKGROUND

Critical care information systems (CCIS) are computer-based systems designed to process the growing amount of complex medical data in intensive care units (ICU). Previous studies have shown that CCICs can increase the quality of patient care by reducing errors and improving work efficiency; however, other studies have shown that CCISs can also cause harmful effects by disrupting workflow, facilitating medication errors or increasing charting time. The factors that decide whether a CCIS has a positive or negative impact on patient care are summarized under the term "usability". This article summarizes the results of three previously published papers on this topic.

OBJECTIVE

The aim of the study was to identify which CCIS functions were considered useful by clinical ICU staff and how well these functions are implemented in the CCISs currently used in German ICUs.

MATERIAL AND METHODS

An online survey was performed targeting nurses and physicians working in German ICUs using a previously validated questionnaire. The questionnaire included a list of functions (36 for physicians/31 for nurses) that were preselected by experts based on a comprehensive model of ICU work processes. Each of these functions was rated by the study participants on a Likert scale ranging from 0 (worst rating) to 5 (best rating) with respect to the usefulness to identify which functions of CCIS can truly be considered as useful by clinical ICU staff. Furthermore, the participants rated how well these functions were implemented in the CCIS currently in use on the ICU, also using a Likert scale of 0-5. Further questions were provided to rate specific technical usability aspects of the CCISs currently in use. In addition, to capture possible confounders the questionnaire recorded 18 individual and workspace characteristics which might influence the ratings.

RESULTS

A total of 171 nurses and 741 physicians participated in the survey of which 535 used CCISs. Of the functions 33 were rated as useful for doctors and 28 functions for nurses with median scores between 4 and 5. Participants currently using CCISs gave higher ratings compared to participants not using CCISs. The quality of the functions was rated relatively lower than the usefulness and the availability. Furthermore, currently used CCISs in Germany differ greatly in their technical and task-specific usability. Of the CCISs investigated, the system ICUData had the best overall rating and technical usability followed by the systems ICM and MetaVision. The same three CCIS were rated best in task-specific functions without significant differences between them.

CONCLUSION

Those functions that were identified as useful based on the ratings of clinical ICU staff should be implemented in current CCIS. The list of these functions might be regarded as a first step towards providing a catalog of functional requirements for CCISs. Furthermore, as the results show that the quality of the available functions was rated lower than the availability of the functions, manufacturers should shift more of the effort away from the development of new features and focus on improving the user-friendliness and quality of existing functions.

The Impact of an Enterprise Electronic Medical Record (EEMR) Model vs a Clinical Information System (CIS) Model on Usability, Efficiency, and Adaptability

This paper describes a methodology that engages the clinical community into the design process of creating Clinical Information Systems (CISs) under a Clinical Team-Led Design (CTLD) approach in the context of using Immediately Adaptable (IA) system development technology. The methodology is contrasted against the Enterprise Electronic Medical Record (EEMR) model for usability, efficiency, and adaptability. The methodology was tested in a Breast Cancer setting where the CIS went through 4 rapid agile stages. Time and motion statistics, training times, system changes, and user feedback data was collected for assessment. The results showed that the Breast CIS increased time efficiency by 30% in the first 3 months of implementation. Users reported high usability and trainability of the system. Over 95% of system design change requests were satisfied with an average turn-around time of 3 days. The results show that systems designed under a CTLD approach, accompanied by Immediately Adaptable system architecture, provide greater efficiency for staff in clinical settings while enabling the workflow processes to be adapted dynamically as part of continuous process improvement.

Evaluation of the USE IT-questionnaire for the Evaluation of the Adoption of Electronic Patient Records by Healthcare Professionals

Background: A combined quantitative and qualitative socio-technical approach is applied in two evaluation studies of electronic patient records (EPR). In these studies the focus was on factors influencing the adoption of the EPR by care providers.

Objective: The research approach is based on the USE IT-model. In addition to the USE IT-interview model, the USE IT-questionnaire is presented and evaluated in order to pre sent a valid and useful integrated approach for the evaluation of IT-adoption in healthcare.

Methods: The USE IT-questionnaire was evaluated by applying a principal component analysis of the quantitative results in two cases (n = 222), and by comparison of the resulting factors with the determinants of the USE IT-model.

Results: The factor analysis of the USE IT-questionnaire resulted in six valid factors: 1. Task support satisfaction, 2. Interface satisfaction, 3. Compatibility, 4. Collaboration, 5. Learnability, and 6. Accessibility. The questions about resources did not combine into a factor.

Conclusion: The detailed questions of the questionnaire lead to decomposition of the constructs Task support satisfaction and Ease of use into factors. The construct Task support satisfaction which was supposed to measure relevance was decomposed in two factors measuring relevance and two factors measuring micro-requirements.

Proposing Electronic Health Record Usability Requirements Based on Enriched ISO 9241 Metric Usability Model

Introduction

System usability assessment is among the important aspects in assessing the quality of clinical information technology, especially when the end users of the system are concerned. This study aims at providing a comprehensive list of system usability.

Methods

This research is a descriptive cross-sectional one conducted using Delphi technique in three phases in 2013. After experts' ideas were concluded, the final version of the questionnaire including 163 items in three phases was presented to 40 users of information systems in hospitals. The grading ranged from 0-4. Data analysis was conducted using SPSS software. Those requirements with a mean point of three or higher were finally confirmed.

Results

The list of system usability requirements for electronic health record was designed and confirmed in nine areas including suitability for the task (24 items), self-descriptiveness (22 items), controllability (19 questions), conformity with user expectations (25 items), error tolerance (21 items), suitability for individualization (7 items), suitability for learning (19 items), visual clarity (18 items) and auditory presentation (8 items).

Conclusion

A relatively comprehensive model including useful requirements for using EHR was presented which can increase functionality, effectiveness and users' satisfaction. Thus, it is suggested that the present model be adopted by system designers and healthcare system institutions to assess those systems.

Investigation of the Usability of Computerized Critical Care Information Systems in Germany

INTRODUCTION

The term "usability" describes how effectively, efficiently, and with what level of user satisfaction an information system can be used to accomplish specific goals. Computerized critical care information systems (CCISs) with high usability increase quality of care and staff satisfaction, while reducing medication errors. Conversely, systems lacking usability can interrupt clinical workflow, facilitate errors, and increase charting time. The aim of this study was to investigate and compare usability across CCIS currently used in Germany.

METHODS

In this study, German intensive care unit (ICU) nurses and physicians completed a specialized, previously validated, web-based questionnaire. The questionnaire assessed CCIS usability based on three rating models: an overall rating of the systems, a model rating technical usability, and a model rating task-specific usability.

RESULTS

We analyzed results from 535 survey participants and compared eight different CCIS commonly used in Germany. Our results showed that usability strongly differs across the compared systems. The system ICUData had the best overall rating and technical usability, followed by the platforms ICM and MetaVision. The same three systems performed best in the rating of task-specific usability without significant differences between each other. Across all systems, overall ratings were more dependent on ease-of-use aspects than on aspects of utility/functionality, and the general scope of the functions offered was rated better than how well the functions are realized.

DISCUSSION

Our results suggest that manufacturers should shift some of their effort away from the development of new features and focus more on improving the ease-of-use and quality of existing features.

Usability problems do not heal by themselves: National survey on physicians' experiences with EHRs in Finland

PURPOSE

Survey studies of health information systems use tend to focus on availability of functionalities, adoption and intensity of use. Usability surveys have not been systematically conducted by any healthcare professional groups on a national scale on a repeated basis. This paper presents results from two cross-sectional surveys of physicians' experiences with the usability of currently used EHR systems in Finland. The research questions were: To what extent has the overall situation improved between 2010 and 2014? What differences are there between healthcare sectors?

METHODS

In the spring of 2014, a survey was conducted in Finland using a questionnaire that measures usability and respondents' user experiences with electronic health record (EHR) systems. The survey was targeted to physicians who were actively doing clinical work. Twenty-four usability-related statements, that were identical in 2010 and 2014, were analysed from the survey. The respondents were also asked to give an overall rating of the EHR system they used. The study data comprised responses from 3081 physicians from the year 2014 and from 3223 physicians in the year 2010, who were using the nine most commonly used EHR system brands in Finland.

RESULTS

Physicians' assessments of the usability of their EHR system remain as critical as they were in 2010. On a scale from 1 ('fail') to 7 ('excellent') the average of overall ratings of their principally used EHR systems varied from 3.2 to 4.4 in 2014 (and in 2010 from 2.5 to 4.3). The results show some improvements in the following EHR functionalities and characteristics: summary view of patient's health status, prevention of errors associated with medication ordering, patient's medication list as well as support for collaboration and information exchange between the physician and the nurses. Even so, support for cross-organizational collaboration between physicians and for physician-patient collaboration were still considered inadequate. Satisfaction with technical features had not improved in four years. The results show marked differences between the EHR system brands as well as between healthcare sectors (private sector, public hospitals, primary healthcare). Compared to responses from the public sector, physicians working in the private sector were more satisfied with their EHR systems with regards to statements about user interface characteristics and support for routine tasks. Overall, the study findings are similar to our previous study conducted in 2010.

CONCLUSIONS

Surveys about the usability of EHR systems are needed to monitor their development at regional and national levels. To our knowledge, this study is the first national eHealth observatory questionnaire that focuses on usability and is used to monitor the long-term development of EHRs. The results do not show notable improvements in physician's ratings for their EHRs between the years 2010 and 2014 in Finland. Instead, the results indicate the existence of serious problems and deficiencies which considerably hinder the efficiency of EHR use and physician's routine work. The survey results call for considerable amount of development work in order to achieve the expected benefits of EHR systems and to avoid technology-induced errors which may endanger patient safety. The findings of repeated surveys can be used to inform healthcare providers, decision makers and politicians about the current state of EHR usability and differences between brands as well as for improvements of EHR usability. This survey will be repeated in 2017 and there is a plan to include other healthcare professional groups in future surveys.

Efficiency achievements from a user-developed real-time modifiable clinical information system

STUDY OBJECTIVE

This investigation was initiated after the introduction of a new information system into the Nepean Hospital Emergency Department. A retrospective study determined that the problems introduced by the new system led to reduced efficiency of the clinical staff, demonstrated by deterioration in the emergency department's (ED's) performance. This article is an investigation of methods to improve the design and implementation of clinical information systems for an ED by using a process of clinical team-led design and a technology built on a radically new philosophy denoted as emergent clinical information systems.

METHODS

The specific objectives were to construct a system, the Nepean Emergency Department Information Management System (NEDIMS), using a combination of new design methods; determine whether it provided any reduction in time and click burden on the user in comparison to an enterprise proprietary system, Cerner FirstNet; and design and evaluate a model of the effect that any reduction had on patient throughput in the department.

RESULTS

The methodology for conducting a direct comparison between the 2 systems used the 6 activity centers in the ED of clerking, triage, nursing assessments, fast track, acute care, and nurse unit manager. A quantitative study involved the 2 systems being measured for their efficiency on 17 tasks taken from the activity centers. A total of 332 task instances were measured for duration and number of mouse clicks in live usage on Cerner FirstNet and in reproduction of the same Cerner FirstNet work on NEDIMS as an off-line system. The results showed that NEDIMS is at least 41% more efficient than Cerner FirstNet (95% confidence interval 21.6% to 59.8%). In some cases, the NEDIMS tasks were remodeled to demonstrate the value of feedback to create improvements and the speed and economy of design revision in the emergent clinical information systems approach. The cost of the effort in remodeling the designs showed that the time spent on remodeling is recovered within a few days in time savings to clinicians. An analysis of the differences between Cerner FirstNet and NEDIMS for sequences of patient journeys showed an average difference of 127 seconds and 15.2 clicks. A simulation model of workflows for typical patient journeys for a normal daily attendance of 165 patients showed that NEDIMS saved 23.9 hours of staff time per day compared with Cerner FirstNet.

CONCLUSION

The results of this investigation show that information systems that are designed by a clinical team using a technology that enables real-time adaptation provides much greater efficiency for the ED. Staff consider that a point-and-click user interface constantly interrupts their train of thought in a way that does not happen when writing on paper. This is partially overcome by the reduction of cognitive load that arises from minimizing the number of clicks to complete a task in the context of global versus local workflow optimization.

Integrated information visualization to support decision making for use of antibiotics in intensive care: design and usability evaluation

Overuse of antibiotics is a critical problem in intensive care today. The situation is further complicated by the extremely data-intensive environment with clinical data presented in distributed, often stand-alone information systems. To access and interpret all data is a complex and time-consuming technical and cognitive challenge. We propose a holistic integrated visualization in the form of a patient overview to support physicians in decision making for use of antibiotics at intensive care units. Special emphasis is put on analysis of work processes to identify information needs, the development of a visualization tool based on an integrated data model, and usability testing of the tool in combination with an eye-tracking technology. The visualization tool was highly rated in terms of user performance and preferences, and the analysis of users' visual patterns showed that different types of data visualization may benefit specialist and resident intensive care physicians depending on the task to be performed. A highly interactive tool for integrated information visualization could potentially increase the understanding of a patient's infection status and ultimately enhance decision making for the use of antibiotics.