User Interface Problems of a Nationwide Inpatient Information System: A Heuristic Evaluation

The Information and Communication Technology Maturity Assessment at Primary Health Care Services Across 9 Provinces in Indonesia: Evaluation Study

Background

Indonesia has rapidly embraced digital health, particularly during the COVID-19 pandemic, with over 15 million daily health application users. To advance its digital health vision, the government is prioritizing the development of health data and application systems into an integrated health care technology ecosystem. This initiative involves all levels of health care, from primary to tertiary, across all provinces. In particular, it aims to enhance primary health care services (as the main interface with the general population) and contribute to Indonesia's digital health transformation.

Objective

This study assesses the information and communication technology (ICT) maturity in Indonesian health care services to advance digital health initiatives. ICT maturity assessment tools, specifically designed for middle-income countries, were used to evaluate digital health capabilities in 9 provinces across 5 Indonesian islands.

Methods

A cross-sectional survey was conducted from February to March 2022, in 9 provinces across Indonesia, representing the country's diverse conditions on its major islands. Respondents included staff from public health centers (Puskesmas), primary care clinics (Klinik Pratama), and district health offices (Dinas Kesehatan Kabupaten/Kota). The survey used adapted ICT maturity assessment questionnaires, covering human resources, software and system, hardware, and infrastructure. It was administered electronically and involved 121 public health centers, 49 primary care clinics, and 67 IT staff from district health offices. Focus group discussions were held to delve deeper into the assessment results and gain more descriptive insights.

Results

In this study, 237 participants represented 3 distinct categories: 121 public health centers, 67 district health offices, and 49 primary clinics. These instances were selected from a sample of 9 of the 34 provinces in Indonesia. Collected data from interviews and focus group discussions were transformed into scores on a scale of 1 to 5, with 1 indicating low ICT readiness and 5 indicating high ICT readiness. On average, the breakdown of ICT maturity scores was as follows: 2.71 for human resources' capability in ICT use and system management, 2.83 for software and information systems, 2.59 for hardware, and 2.84 for infrastructure, resulting in an overall average score of 2.74. According to the ICT maturity level pyramid, the ICT maturity of health care providers in Indonesia fell between the basic and good levels. The need to pursue best practices also emerged strongly. Further analysis of the ICT maturity scores, when examined by province, revealed regional variations.

Conclusions

The maturity of ICT use is influenced by several critical components. Enhancing human resources, ensuring infrastructure, the availability of supportive hardware, and optimizing information systems are imperative to attain ICT maturity in health care services. In the context of ICT maturity assessment, significant score variations were observed across health care levels in the 9 provinces, underscoring the diversity in ICT readiness and the need for regionally customized follow-up actions.

Integrated Diabetes Self-Management (IDSM) mobile application to improve self-management and glycemic control among patients with Type 2 Diabetes Mellitus (T2DM) in Indonesia: A mixed methods study protocol

The number of patients with diabetes in Indonesia reach 19,47 million in 2021, mostly is caused by the adoption of sedentary and unhealthy lifestyles. Continuous self-management is important in diabetes care. It requires optimal coordination and communication between patients, families, and health care provider. The use of communication technology could be solution to the problem. This study aims to initiate an android-based mobile apps technology as a tool for patient, family, and healthcare provider to optimize patient with T2DM treatment. This study will be conducted in Public Health Centers (PHCs) in Yogyakarta using an exploratory sequential mixed-methods design which is divided into three phases. The first phase will use qualitative descriptive methods. Patients with T2DM, families, nurses, physicians and Health Social Security Agency officers in Indonesia will be involved in a focus group discussion (FGD) and in-depth interviews to understand their needs in optimizing the treatment. The second phase will be the development of an android-based application on the first phase results. The apps will be usability tested by involving experts (heuristic evaluation) and users (think aloud method) to ensure that the apps really meet targeted user's need. In the third phase, we will collect feedback from user after using this apps for three months. The effectiveness of the apps will be measured by self-management improvement and glycemic control of patients with T2DM. The non-equivalent control group design will be applied using a pre-repeated post-test control group. The result of this study will be an Android-based Application which will be called Integrated Diabetes Self-Management (IDSM) app to optimize the implementation of diabetes self-management which can improve glycemic control of patients with T2DM as one of the indicators of the Indonesian Chronic Disease Management Program at PHCs.

Digital Future of Emergency Medical Services: Envisioning and Usability of Electronic Patient Care Report System

Despite the efforts of emerging technologies in the healthcare system, there is still a slower rate of acceleration in prehospital settings compared with the hospitals in digital transformation adaptation. The acknowledgment that digital transformation is significant to healthcare is reflected in planning for the future of digital healthcare. Thus, this study aimed to measure the usability of the electronic patient care report (ePCR) system among emergency medical services (EMS) staff who work in prehospital settings. A descriptive cross-sectional correlation study was used. Two hundred fifty EMS staff who are working in the prehospital setting at Saudi Red Crescent Authority in the Kingdom of Saudi Arabia were surveyed, and the response rate was 79.2% (198). An adapted tool of the Computer System Usability Questionnaire survey was used to collect data. The data were coded numerically and subjected to descriptive and inferential statistical analysis including Pearson’s correlation coefficient using the statistical software (SPSS 21). The majority of the participants rate their ePCR system as “useable” at a high level with a score of 3.41 (SD = 1.021). The overall mean of the ePCR system’s three subscales: system usefulness, information quality, interface quality, and overall satisfaction were 3.39 (SD = 1.152), 3.30 (SD = 1.052), 3.57 (SD = 1.064), and 3.37 (SD = 1.239), respectively. The least liked aspect of ePCR system software was information quality 81 (40.9%). Furthermore, there was a significant correlation between the age of EMS staff and the usability of the ePCR system (r = −0.150 ${\hspace{0.17em}}^{\ast }$ , $P=0.035$ ). The results suggest that healthcare institutions’ policy and decision-makers pay close attention to performing standardized training for the staff on their ePCR system before going to the field to increase efficiency and productivity. Furthermore, the users in this study identified other system features that, if included, could have enhanced usability, and improved functions and capabilities of the design to meet the EMS staff’s expectations.

Evaluating the usability of a national health information system with heuristic method

BACKGROUND

Hospital Statistics and Information System is one of the most important health information systems in Iran used in all hospitals in this country. Usability problems can reduce the speed and precision of users when interacting with this system. This study aimed to identify the usability problems of a national health system called "AVAB".

MATERIALS AND METHODS

This descriptive cross-sectional study was conducted in 2020, and three experts evaluated the usability of this system independently by the heuristic evaluation method. Nielsen's usability principles were used to identify usability problems and to classify their severity.

RESULTS

A total of 86 unique problems were identified. The highest number of problems were related to the two principles of "help and documentation" and "match between system and the real world" with 23 and 11 usability problems, respectively. The lowest number of problems were related to the two principles of "visibility of system status" and "help users recognize, diagnose, and recover from errors," each with three problems. 58.1% of the identified problems were in the group of major and catastrophic problems.

CONCLUSIONS

With the help of heuristic evaluation method, a significant number of usability problems of Hospital Statistics and Information System were identified. Most of the identified problems were major and catastrophic, and it is necessary to solve these problems by the designers and developers of this system.

Comparison of usability evaluation methods for a health information system: heuristic evaluation versus cognitive walkthrough method

Background

There are differences of opinion regarding the selection of the most practical usability evaluation method among different methods. The present study aimed to compare two expert-based evaluation methods in order to assess a nursing module as the most widely used module of a Hospital Information System (HIS).

Methods

Five independent evaluators used the Heuristic Evaluation (HE) and Cognitive Walkthrough (CW) methods to evaluate the nursing module of Shafa HIS. In this regard, the number and severity of the recognized problems according to the usability attributes were compared using two evaluation methods.

Results

The HE and CW evaluation methods resulted in the identification of 104 and 24 unique problems, respectively, of which 33.3% of recognized problems in the CW evaluation method overlapped with the HE method. The average severity of the recognized problems was considered to be minor (2.34) in the HE method and major (2.77) in the CW evaluation method. There was a significant difference in terms of the total number and average severity of the recognized problems by these methods (P < 0.001). Based on the usability attribute, the HE method identified a larger number of problems concerning all usability attributes, and a significant difference was observed in terms of the number of recognized problems in both methods for all attributes except ‘memorability’. Also, there was a significant difference between the two methods based on the average severity of recognized problems only in terms of ‘learnability’.

Conclusion

The HE method identified more problems with lower average severity while the CW was able to recognize fewer problems with higher average severity. Regarding the evaluation goal, the HE method was able to be used to improve the effectiveness and satisfaction of the HIS. Furthermore, the CW evaluation method is recommended to identify usability problems with the highest average severity, especially in terms of ‘learnability’.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01905-7.

Identification and Classification of Usability Problems in a Nursing Information System: A Heuristic Evaluation

Nursing information system is a system that nurses are required to use in order to perform their daily activities. Poor user interface design of nursing information system can cause problems in nurses' interaction with the system. This research aimed to evaluate the usability of nursing information system. During this study, five evaluators examined the nursing information system with the heuristic evaluation method and by using the checklist of Nielsen usability principles. Then, the identified problems were categorized into 10 Nielsen usability principles, and the severity of the problems was determined. They also attributed each problem to one of usability attributes. A total of 104 unique problems were identified. Most of the problems were related to the principle of "consistency and standards." More than a third of the identified problems were classed as major and catastrophe, with the highest severity in the average problems of "help and documentation" and "error prevention." Most of the identified problems were attributed to effectiveness and satisfaction. Because a significant number of identified usability problems in nursing information system were major and catastrophe and were attributed to effectiveness and satisfaction, being able to remedy the problems could improve nurses' interaction with the system user interface and increase satisfaction and effectiveness of nurses.

Usability evaluation of a nursing information system by applying cognitive walkthrough method

BACKGROUND AND OBJECTIVE

The user interface usability of the nursing information system (NIS) should be such that the nurses can learn and interact with it easily and quickly. Therefore, it is necessary to identify and solve the usability problems of these systems. The present study aimed to evaluate the usability of a NIS using the cognitive walkthrough (CW) evaluation method.

METHODS

Based on five selected scenarios, five evaluators evaluated the NIS in Shafa Hospital Information System. After identifying the problems, the evaluators assigned each problem to one of the usability attributes. The severity of each identified problem was determined by the evaluators and five real users of the system.

RESULTS

In total, 24 unique problems were identified. The average severity of the problems was determined by the evaluators (2.77) and the actual users (2.82) in the "major problem" category. The highest number of problems were assigned to the scenarios 3 and 2 with 15 and 14 problems, respectively. The highest average severity in terms of evaluations and actual users was related to the scenario 5 (3.06 and 2.94, respectively), which was in the "major problem" category. The highest number of problems were associated with learnability (8 problems) and efficiency (6 problems).

CONCLUSION

Since most of the nurses do not have enough time for learning the system, and given that the selected scenarios for evaluation were based on the most frequent daily tasks that nurses performed using the NIS; The correction of usability problems in the scenarios 3 and 2 that had the highest number of problems and the scenario 5 that had the highest average severity and execution time; it can reduce the nurses' cognitive load and the learning time of the system and increase the efficiency of nurses.

Usability evaluation of selected picture archiving and communication systems at the national level: Analysis of users' viewpoints

BACKGROUND AND OBJECTIVES

Picture Archiving and Communication System (PACS) can be considered as one of the most important information systems used in healthcare facilities and its usability problems can lead to delays in the acceptance of information systems, increased medical errors and also user dissatisfaction. The primary purpose of this study was to evaluate the usability of PACS from the perspective of its main users.

METHODS

This study used a combination of qualitative and quantitative methods and was carried out in 2019 where the research community consisted of PACS in five selected companies in Iran. The statistical sample included 200 individuals using the PACS in several hospitals across the country. Moreover, the sample was selected using the multistage random method. The data were then collected using the standard Computer System Usability Questionnaire (CSUQ) consisting of 5 sections and 19 items. Finally, the data were analyzed by SPSS software, version 18 using both descriptive and inferential methods. Content analysis was done for the qualitative data sets.

RESULTS

It was indicated that ease of use, as a category, was not significantly different from the perspective of various PACS users. However, the ANOVA test revealed that there were significant differences in terms of information quality, user interface quality, overall user satisfaction and usability of PACS from the users' perspectives. Furthermore, content analysis of users' comments showed that speeding up the image processing and frequent system failures were amongst the most positive and negative aspects of the PACS, respectively.

CONCLUSIONS

According to the perspective of the users of the investigated PACS in Iran, the usability of these PACS had a favorable status regarding ease of use while provided lower information quality. Generally, based on the users' viewpoints, the PACS from Company B were the most usable while the PACS provided by Company D were the least usable. It is suggested that the information quality and user interface of systems be improved by using appropriate analysis and needs assessment of the end users.

A combination of two methods for evaluating the usability of a hospital information system

Background

None of the evaluation methods can identify all the usability problems of information systems. So far, no study has sufficiently investigated the potential of a combination of these methods to identify usability problems. The present study aimed at examining the potential for combining two commonly utilized user-based and expert-based methods to evaluate the usability of a hospital information system.

Methods

Think aloud (TA) and Heuristic evaluation (HE) methods were used to identify the usability problems of two subsystems of the Social Security Electronic System in Iran. To this end, the problems were categorized into five groups based on ISO-Nielsen usability attributes. The Chi-square test was applied to compare the intended methods based on the total number of problems and the number of problems within each group, followed by utilizing the Mann-Whitney U test to compare the mean severity scores of these methods.

Results

The evaluation by combining these methods yielded 423 problems of which 75% varied between the methods. The two methods were significantly different in terms of the total number of problems, the number of problems in each usability group, and the mean severity of two satisfaction and efficiency attributes (P < 0.05). However, no significant difference was observed between the two methods based on the mean severity of problems and severity scores related to three usability attributes i.e., effectiveness, learnability, and error prevention (P > 0.05). In addition, the mean severity of problems identified by each method was at the “Major” level.

Conclusion

Based on the results, although the mean severity scores of the identified problems were not significantly different, these methods identify heterogeneous problems. HE mainly identifies problems related to satisfaction, learnability, and error prevention while TA detects problems related to effectiveness and efficiency attributes. Therefore, using a combination of these two methods can identify a wider range of usability problems.

The relationship between user interface problems of an admission, discharge and transfer module and usability features: a usability testing method

Background

The admission, discharge and transfer (ADT) module is used in the hospital information system (HIS) for the purposes of managing appointments, patient admission, daily control of hospital beds, planning surgery procedures, keeping up-to-date on patient discharges, and registering patient transfers within or outside the hospital. The present study aimed to evaluate the usability of ADT module of a HIS through usability testing and assess the relationship between the number of user interface problems and usability features (i.e. effectiveness, efficiency, and satisfaction).

Methods

This descriptive analytical study was conducted in Shahid Beheshti hospital in Kashan, Iran, in 2017. The participating users were eight students in their last semester of a Bachelor of Health Information Technology Sciences degree. First, the users were introduced to the module functions in a two-hour session; ten days later, the users were asked to perform scenarios designed based on seven tasks and take notes of the problems encountered in performing each task after it was over. Effectiveness was measured based on the rate of completing the tasks, efficiency based on the time taken to perform each task, and satisfaction based on the users’ answers to a satisfaction questionnaire. The relationship between these three usability features and the number of problems noted was assessed using Spearman’s test in SPSS version 16.

Results

Thirteen unique usability problems were identified from the perspective of the users. Effectiveness was rated as 58.9%, efficiency as 53.3%, and mean user satisfaction as 53.4 ± 10.6. The number of problems in each task had significant relationships to the effectiveness (P = 0.009) and efficiency (P = 0.016) scores. User satisfaction also had a significant relationship with the effectiveness (P = 0.043) but not with the efficiency (P = 0.230) scores.

Conclusions

In the view of the potential users, a HIS, used in more than 200 hospitals in a developing country, has several usability problems in its ADT module and its effectiveness, efficiency, and user satisfaction were not acceptable. The number of usability problems in the HIS user interface affected the effectiveness, efficiency and user satisfaction of the system.

Usability Evaluation of Three Admission and Medical Records Subsystems Integrated into Nationwide Hospital Information Systems: Heuristic Evaluation

Introduction:

Usability is one of the quality criteria for information systems and its weakness is one of the main barriers to the adoption of these systems. The purpose of this study was to evaluate the usability of admission and medical records module of three widely used hospital information systems (HISs).

Methods:

In this descriptive study the usability of admission and medical records module of three HISs (HIS1, HIS2, and HIS3) was evaluated using heuristic evaluation method. For each HIS, three expert users of the same system assessed the user interface independently, completed a usability evaluation checklist, and rated severity of each identified problem. The checklist was based on Nielsen’s heuristics. For each HIS, three heuristics that have the highest and lowest problem rates and greatest severity of problems were categorized into three separate groups. The results were analyzed using descriptive statistics.

Results:

Although HIS1 and HIS2 were used in more hospitals than HIS3, the results showed that the usability problem rates of them were significantly higher than HIS3. The heuristics of “help and documentation”, “flexibility and efficiency of use”, and “visibility of system status” in the three HISs were categorized into the “highest rate of problems”, “lowest rate of problems”, and “highest severity of problems” groups, respectively. The heuristics of “diagnose and recover from errors”, “error prevention”, and “help and documentation” in HIS1 and HIS2 were categorized into the “highest rate of problems” group.

Conclusions:

The results of this study and previous studies show that the most common usability problems with HISs are related to heuristics of “help and documentation”, “error prevention”, and “help users recognize, diagnose and recover from errors.” Also, the large number of hospitals using one HIS does not demonstrate its high usability to others.

Determinants of user acceptance of a specific social platform for older adults: An empirical examination of user interface characteristics and behavioral intention

The use of the Internet and social applications has many benefits for the elderly, but numerous investigations have shown that the elderly do not perceive online social networks as a friendly social environment. Therefore, TreeIt, a social application specifically designed for the elderly, was developed for this study. In the TreeIt application, seven mechanisms promoting social interaction were designed to allow older adults to use social networking sites (SNSs) to increase social connection, maintain the intensity of social connections and strengthen social experience. This study's main objective was to investigate how user interface design affects older people's intention and attitude related to using SNSs. Fourteen user interface evaluation heuristics proposed by Zhang et al. were adopted as the criteria to assess user interface usability and further grouped into three categories: system support, user interface design and navigation. The technology acceptance model was adopted to assess older people's intention and attitude related to using SNSs. One hundred and one elderly persons were enrolled in this study as subjects, and the results showed that all of the hypotheses proposed in this study were valid: system support and perceived usefulness had a significant effect on behavioral intention; user interface design and perceived ease of use were positively correlated with perceived usefulness; and navigation exerted an influence on perceived ease of use. The results of this study are valuable for the future development of social applications for the elderly.