Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method

Evaluating the usability of a cancer registry system using Cognitive Walkthrough, and assessing user agreement with its problems

OBJECTIVE/AIM

Good design of cancer registry systems makes them easy to use, while poor design of their user interfaces leads to user dissatisfaction and resistance. The objective of this study was to evaluate the usability of a cancer registry system using Cognitive Walkthrough (CW) and to assess users' agreement with its usability problems.

METHODS

CW was used to evaluate the registry system. We developed a checklist to help evaluators speed up the evaluation process, a problems form to collect the usability issues identified by the evaluators, and a problems severity form to determine the severity of problems by the evaluators. The problems were classified into two categories according to the CW questions and the system tasks. The agreement of the users with the system problems was examined by an online questionnaire. Users' agreement with the problems was then analyzed using the Interclass Correlation Coefficient in the SPSS 22 (Statistical Package for Social Science).

RESULTS

In this study, 114 problems were identified. In the categorization of problems based on the CW questions, 41% (n = 47) of the problems concerned the issue of "users do not know what to do at each stage of working with the system", 24% (n = 27) were classified as "users cannot link what they intend to do with system controls", and 22% (n = 25) were related to "user's lack of understanding of the system processes". Based on user tasks, about 36% (n = 41) of the problems were related to "removing patient duplication" and 33% (n = 38) were related to "registration of patient identification information". User agreement with the problems was high (CI 95% = 0.9 (0.96, 0.98)).

CONCLUSION

System problems often originate from user ignorance about what to do at each stage of using the system. Also, half of the system problems concern a mismatch between what users want to do and the system controls, or a lack of understanding about what the system does at different stages. Therefore, to avoid user confusion, designers should use clues and guides on the screen for users, design controls consistent with the user model of thinking, and provide appropriate feedback after each user action to help users understand what the system is doing. The high agreement of users with the problems showed that in the absence of users system designers can use CW to identify the problems that users face in the real environment.

Agile, Easily Applicable, and Useful eHealth Usability Evaluations: Systematic Review and Expert-Validation

Background  Electronic health (eHealth) usability evaluations of rapidly developed eHealth systems are difficult to accomplish because traditional usability evaluation methods require substantial time in preparation and implementation. This illustrates the growing need for fast, flexible, and cost-effective methods to evaluate the usability of eHealth systems. To address this demand, the present study systematically identified and expert-validated rapidly deployable eHealth usability evaluation methods.

Objective  Identification and prioritization of eHealth usability evaluation methods suitable for agile, easily applicable, and useful eHealth usability evaluations.

Methods  The study design comprised a systematic iterative approach in which expert knowledge was contrasted with findings from literature. Forty-three eHealth usability evaluation methods were systematically identified and assessed regarding their ease of applicability and usefulness through semi-structured expert interviews with 10 European usability experts and systematic literature research. The most appropriate eHealth usability evaluation methods were selected stepwise based on the experts' judgements of their ease of applicability and usefulness.

Results  Of these 43 eHealth usability evaluation methods identified as suitable for agile, easily applicable, and useful eHealth usability evaluations, 10 were recommended by the experts based on their usefulness for rapid eHealth usability evaluations. The three most frequently recommended eHealth usability evaluation methods were Remote User Testing, Expert Review, and Rapid Iterative Test and Evaluation Method. Eleven usability evaluation methods, such as Retrospective Testing, were not recommended for use in rapid eHealth usability evaluations.

Conclusion  We conducted a systematic review and expert-validation to identify rapidly deployable eHealth usability evaluation methods. The comprehensive and evidence-based prioritization of eHealth usability evaluation methods supports faster usability evaluations, and so contributes to the ease-of-use of emerging eHealth systems.

Clinical decision support for intervention reduction in neonatal patients: A usability assessment

Objective

This study investigated how effectively simplified cognitive walkthroughs, performed independently by four nonclinical researchers, can be used to assess the usability of clinical decision support software. It also helped illuminate the types of usability issues in clinical decision support software tools that cognitive walkthroughs can identify.

Method

A human factors professor and three research assistants each conducted an independent cognitive walkthrough of a web-based demonstration version of T3, a physiologic monitoring system featuring a new clinical decision support software tool called MAnagement Application (MAP). They accessed the demo on personal computers in their homes and used it to walk through several pre-specified tasks, answering three standard questions at each step. Then they met to review and prioritize the findings.

Results

Evaluators acknowledged several positive features including concise, helpful tooltips and an informative column in the patient overview which allows users direct (one-click) access to protocol eligibility and compliance criteria. Recommendations to improve usability include: modify the language to clarify what user actions are possible; visually indicate when eligibility flags are snoozed; and specify which protocol's data is currently being shown.

Conclusion

Independent, simplified cognitive walkthroughs can help ensure that clinical decision support software tools will appropriately support clinicians. Four researchers used this technique to quickly, inexpensively, and effectively assess T3's new MAP tool, which suggests positive actions, such as removing a patient from a ventilator. Results indicate that, while there is room for usability improvements, the MAP tool may help reduce clinician's cognitive load, facilitating improved care. The study also confirmed that cognitive walkthroughs identify issues that make clinical decision support software hard to learn or remember to use.