Usability evaluation of a nursing information system by applying cognitive walkthrough method

Usability and Evaluation of a Health Information System in the Emergency Department: Mixed Methods Study

BACKGROUND

A lack of information during an emergency visit leads to the experience of powerlessness for patients and their family members, who may also feel unprepared to cope with acute symptoms. The ever-changing nature and fast-paced workflow in the emergency department (ED) often affect how health care professionals can tailor information and communication to the needs of the patient.

OBJECTIVE

This study aimed to evaluate the usability and experience of a newly developed information system. The system was developed together with patients and their family members to help provide the information needed in the ED.

METHODS

We conducted a mixed methods study consisting of quantitative data obtained from the System Usability Scale questionnaire and qualitative interview data obtained from purposively selected participants included in the quantitative part of the study.

RESULTS

A total of 106 patients and 14 family members (N=120) answered the questionnaire. A total of 10 patients and 3 family members participated in the interviews. Based on the System Usability Scale score, the information system was rated close to excellent, with a mean score of 83.6 (SD 12.8). Most of the participants found the information system easy to use and would like to use it again. The participants reported that the system helped them feel in control, and the information was useful. Simplifications were needed to improve the user experience for the older individuals.

CONCLUSIONS

This study demonstrates that the usability of the information system is rated close to excellent. It was perceived to be useful as it enabled understanding and predictability of the patient's trajectory in the ED. Areas for improvement include making the system more usable by older individuals. The study provides an example of how a technological solution can be used to diminish the information gap in an ED context.

Heuristic smartphone usability evaluations of the mobile application NANDA, nursing interventions classification, and nursing outcomes classification customized for nursing home registered nurses

PURPOSE

We aimed to evaluate the usability of a smartphone application consisting of standardized nursing language (SNL) using NANDA, Nursing Intervention Classification, and Nursing Outcome Classification for nursing home nurses.

DATA SOURCES

Applying convenience sampling, a total of 14 experts and 15 real users were invited to test and evaluate the smartphone application independently. For the usability evaluation of the developed application, the Korean version of the Mobile Application Rating Scale for experts and Mobile Application Rating Scale: User Version developed by Stoyanov et al. were used.

DATA SYNTHESIS

Both groups determined that the SNL application was quite informative about SNL and efficient function; however, the engagement was quite lower than other categories.

CONCLUSIONS

Although SNLs were scientifically developed for several decades, the widely available technological application for registered nurses in different languages is urgently needed to improve quality of nursing care.

IMPLICATIONS FOR NURSING PRACTICE

The identified problems and recommendations by users and experts using heuristic evaluation will be reflected in the application's final version to be used for research.

How do automated reasoning features impact the usability of a clinical task management system? Development and usability testing of a prototype

BACKGROUND

Electronic clinical task management systems (ECTMSs) have been developed and adopted by care providers to improve care coordination. Some systems utilised automated reasoning (AR) to enable more intelligent task management functionalities, such as automated task allocation. Yet, the impact of such features on usability remains unclear. Poor usability of health information systems has been described to cause frustration and contribute to patient safety incidents.

AIM

To design AR features for an ECTMS and to evaluate their impact on usability.

METHODS

In this mixed methods study, four ECTMS feature prototypes were co-designed with two clinicians. For each prototype, one AR variant and one non-AR variant with equivalent functionalities were developed. A moderated usability testing was conducted with seven clinicians to obtain ease-of-use ratings of prototypes and measure task durations. Parameters related to demographics and attitudes of participants were obtained via a questionnaire. A framework analysis was performed to summarise qualitative feedback. To determine statistical relationships of study variables, Spearmańs rank coefficients were calculated and presented as a correlation matrix.

RESULTS

Three out of four prototypes received higher median ease-of-use ratings for AR variants and were associated with shorter average task durations. Multiple clinical use cases suitable for AR were identified. Preference for AR was found to moderately correlate with digital proficiency and prior experience with ECTMSs. Insufficient trust in automation, alert fatigue, and system customisation were identified as challenges in the adoption of AR features.

CONCLUSIONS

This study provides evidence for the potential of AR to enhance usability in ECTMSs. Consideration of psychological and organisational context of users in the feature design was found to be decisive for usability. Future research should explore implications for operational and clinical outcomes.

Evaluating the usability of a cancer registry system using Cognitive Walkthrough, and assessing user agreement with its problems

OBJECTIVE/AIM

Good design of cancer registry systems makes them easy to use, while poor design of their user interfaces leads to user dissatisfaction and resistance. The objective of this study was to evaluate the usability of a cancer registry system using Cognitive Walkthrough (CW) and to assess users' agreement with its usability problems.

METHODS

CW was used to evaluate the registry system. We developed a checklist to help evaluators speed up the evaluation process, a problems form to collect the usability issues identified by the evaluators, and a problems severity form to determine the severity of problems by the evaluators. The problems were classified into two categories according to the CW questions and the system tasks. The agreement of the users with the system problems was examined by an online questionnaire. Users' agreement with the problems was then analyzed using the Interclass Correlation Coefficient in the SPSS 22 (Statistical Package for Social Science).

RESULTS

In this study, 114 problems were identified. In the categorization of problems based on the CW questions, 41% (n = 47) of the problems concerned the issue of "users do not know what to do at each stage of working with the system", 24% (n = 27) were classified as "users cannot link what they intend to do with system controls", and 22% (n = 25) were related to "user's lack of understanding of the system processes". Based on user tasks, about 36% (n = 41) of the problems were related to "removing patient duplication" and 33% (n = 38) were related to "registration of patient identification information". User agreement with the problems was high (CI 95% = 0.9 (0.96, 0.98)).

CONCLUSION

System problems often originate from user ignorance about what to do at each stage of using the system. Also, half of the system problems concern a mismatch between what users want to do and the system controls, or a lack of understanding about what the system does at different stages. Therefore, to avoid user confusion, designers should use clues and guides on the screen for users, design controls consistent with the user model of thinking, and provide appropriate feedback after each user action to help users understand what the system is doing. The high agreement of users with the problems showed that in the absence of users system designers can use CW to identify the problems that users face in the real environment.

Visualization Method of Key Knowledge Points of Nursing Teaching Management System Based on SOM Algorithm and Biomedical Diagnosis

The traditional nursing teaching knowledge point recommendation algorithm based on collaborative filtering is difficult to deal with the problem of data sparsity, while the traditional recommendation algorithm based on matrix decomposition has poor scalability in dealing with high-dimensional data, and their recommendation results are only determined according to the prediction score, resulting in low recommendation accuracy. In view of this, a nursing teaching knowledge point recommendation method based on a SOM neural network and ranking factor decomposition machine is proposed. Firstly, the SOM neural network is used to cluster users based on users' academic background information, then the partial order relationship of nursing teaching knowledge points is constructed by using users' explicit and implicit web access behavior, and finally, the factor decomposition machine is used as the ranking function to classify users' academic background web access behavior, borrowing nursing teaching introduction text, and other characteristic information were modeled, and the peer-to-peer ranking learning algorithm was used to accurately recommend nursing teaching knowledge points. Experimental results show that the proposed method can effectively alleviate the problem of data sparsity and improve the accuracy and efficiency of recommendations.

Comparison of usability evaluation methods for a health information system: heuristic evaluation versus cognitive walkthrough method

Background

There are differences of opinion regarding the selection of the most practical usability evaluation method among different methods. The present study aimed to compare two expert-based evaluation methods in order to assess a nursing module as the most widely used module of a Hospital Information System (HIS).

Methods

Five independent evaluators used the Heuristic Evaluation (HE) and Cognitive Walkthrough (CW) methods to evaluate the nursing module of Shafa HIS. In this regard, the number and severity of the recognized problems according to the usability attributes were compared using two evaluation methods.

Results

The HE and CW evaluation methods resulted in the identification of 104 and 24 unique problems, respectively, of which 33.3% of recognized problems in the CW evaluation method overlapped with the HE method. The average severity of the recognized problems was considered to be minor (2.34) in the HE method and major (2.77) in the CW evaluation method. There was a significant difference in terms of the total number and average severity of the recognized problems by these methods (P < 0.001). Based on the usability attribute, the HE method identified a larger number of problems concerning all usability attributes, and a significant difference was observed in terms of the number of recognized problems in both methods for all attributes except ‘memorability’. Also, there was a significant difference between the two methods based on the average severity of recognized problems only in terms of ‘learnability’.

Conclusion

The HE method identified more problems with lower average severity while the CW was able to recognize fewer problems with higher average severity. Regarding the evaluation goal, the HE method was able to be used to improve the effectiveness and satisfaction of the HIS. Furthermore, the CW evaluation method is recommended to identify usability problems with the highest average severity, especially in terms of ‘learnability’.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01905-7.

Clinical decision support for intervention reduction in neonatal patients: A usability assessment

Objective

This study investigated how effectively simplified cognitive walkthroughs, performed independently by four nonclinical researchers, can be used to assess the usability of clinical decision support software. It also helped illuminate the types of usability issues in clinical decision support software tools that cognitive walkthroughs can identify.

Method

A human factors professor and three research assistants each conducted an independent cognitive walkthrough of a web-based demonstration version of T3, a physiologic monitoring system featuring a new clinical decision support software tool called MAnagement Application (MAP). They accessed the demo on personal computers in their homes and used it to walk through several pre-specified tasks, answering three standard questions at each step. Then they met to review and prioritize the findings.

Results

Evaluators acknowledged several positive features including concise, helpful tooltips and an informative column in the patient overview which allows users direct (one-click) access to protocol eligibility and compliance criteria. Recommendations to improve usability include: modify the language to clarify what user actions are possible; visually indicate when eligibility flags are snoozed; and specify which protocol's data is currently being shown.

Conclusion

Independent, simplified cognitive walkthroughs can help ensure that clinical decision support software tools will appropriately support clinicians. Four researchers used this technique to quickly, inexpensively, and effectively assess T3's new MAP tool, which suggests positive actions, such as removing a patient from a ventilator. Results indicate that, while there is room for usability improvements, the MAP tool may help reduce clinician's cognitive load, facilitating improved care. The study also confirmed that cognitive walkthroughs identify issues that make clinical decision support software hard to learn or remember to use.