1
|
Marcilly R, Zheng WY, Quindroit P, Pelayo S, Berdot S, Charpiat B, Corny J, Drouot S, Frery P, Leguelinel-Blache G, Mondet L, Potier A, Robert L, Ferret L, Baysari M. Comparison of the validity, perceived usefulness, and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability. Int J Med Inform 2023; 175:105091. [PMID: 37182411 DOI: 10.1016/j.ijmedinf.2023.105091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Revised: 05/05/2023] [Accepted: 05/06/2023] [Indexed: 05/16/2023]
Abstract
OBJECTIVE Two tools are currently available in the literature to evaluate the usability of medication alert systems, the instrument for evaluating human factors principles in medication-related decision support alerts (I-MeDeSA) and the tool for evaluating medication alerting systems (TEMAS). This study aimed to compare their convergent validity, perceived usability, usefulness, strengths, and weaknesses, as well as users' preferences. METHOD To evaluate convergent validity, two experts mapped TEMAS' items against I-MeDeSA's items with respect to the usability dimensions they target. To assess perceived usability, usefulness, strengths, and weaknesses of both tools, staff with expertise in their medication alerting system were asked to use French versions of the TEMAS and I-MeDeSA. After the use of each tool, participants were asked to complete the System Usability Scale (SUS) and answer questions about the understandability and usefulness of each tool. Finally, participants were asked to name their preferred tool. Numeric scores were statistically compared. Free-text responses were analyzed using an inductive approach. RESULTS Forty-five participants from 10 hospitals took part in the study. In terms of convergent validity, I-MeDeSA focuses more on the usability of the graphical user interface while TEMAS considers a wider range of usability principles. Both tools have a fair level of perceived usability (I-MeDeSA' SUS score = 61.85 and TEMAS' SUS score = 62.87), but results highlight that revisions are necessary to both tools to improve their usability. Participants found TEMAS more useful than I-MeDeSA (t = -3.63, p =.005) and had a clear preference for TEMAS to identify problems in formative evaluation (39 of 45; 0.867, p <.001) and to compare the usability of alert systems during the procurement process (36 of 45; 0.8, p <.001). CONCLUSIONS The TEMAS is perceived as more useful and is preferred by participants. The I-MeDeSA seems more relevant for quick evaluations that focus on the graphical user interface. The TEMAS seems to be more suitable for in-depth usability evaluations of alert systems. Even if both tools are perceived to be equally usable, they suffer from wording, instructional, and organizational problems that hinder their use. The results of this study will be used to improve the design of I-MeDeSA and TEMAS.
Collapse
Affiliation(s)
- Romaric Marcilly
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France; Inserm, CIC-IT 1403, F-59000 Lille, France.
| | - Wu-Yi Zheng
- Black Dog Institute, Randwick, NSW, Australia.
| | - Paul Quindroit
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France.
| | - Sylvia Pelayo
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France; Inserm, CIC-IT 1403, F-59000 Lille, France.
| | - Sarah Berdot
- Assistance Publique - Hôpitaux de Paris, Département de Pharmacie, Hôpital Européen Georges-Pompidou, Paris, France; Inserm, Cordeliers Research Centre, Université de Paris, Sorbonne Université, Paris, France; HeKA, Inria, Paris, France.
| | - Bruno Charpiat
- Pharmacie, Hôpital de la Croix-Rousse, Hospices Civils de Lyon, 69004 Lyon, France.
| | - Jennifer Corny
- Service de Pharmacie, Groupe Hospitalier Paris Saint-Joseph, Paris, France.
| | - Sylvain Drouot
- Clinical Pharmacy Department, Hôpital Bicêtre, APHP, Paris, France.
| | | | - Géraldine Leguelinel-Blache
- Desbrest Institute of Epidemiology and Public Health, Univ Montpellier, Inserm, Montpellier, France; Department of Pharmacy, CHU Nîmes, Univ Montpellier, Nîmes, France.
| | - Lisa Mondet
- Department of Pharmacy, CHU Amiens-Picardie, Amiens, France.
| | - Arnaud Potier
- Service de pharmacie, CH de Luneville, 54300 Luneville, France; Service de pharmacie, CHRU de Nancy, 54000 Nancy, France.
| | - Laurine Robert
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France; CHU Lille, Institut de Pharmacie, Lille, France.
| | - Laurie Ferret
- Department of Pharmacy, General hospital of Valenciennes, 59300, France.
| | - Melissa Baysari
- The University of Sydney, Faculty of Medicine and Health, School of Medical Sciences, Biomedical Informatics and Digital Health, Sydney, Australia.
| |
Collapse
|
2
|
Marcilly R, Zheng WY, Beuscart R, Baysari MT. Comparison of the validity, perceived usefulness and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability: a study protocol. BMJ Open 2021; 11:e050448. [PMID: 34353806 PMCID: PMC8344302 DOI: 10.1136/bmjopen-2021-050448] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Abstract
INTRODUCTION Research has shown that improvements to the usability of medication alert systems are needed. For designers and decisions-makers to assess usability of their alert systems, two paper-based tools are currently available: the instrument for evaluating human-factors principles in medication-related decision support alerts (I-MeDeSA) and the tool for evaluating medication alerting systems (TEMAS). This study aims to compare the validity, usability and usefulness of both tools to identify their strengths and limitations and assist designers and decision-makers in making an informed decision about which tool is most suitable for assessing their current or prospective system. METHODS AND ANALYSIS First, TEMAS and I-MeDeSA will be translated into French. This translation will be validated by three experts in human factors. Then, in 12 French hospitals with a medication alert system in place, staff with expertise in the system will evaluate their alert system using the two tools successively. After the use of each tool, participants will be asked to fill in the System Usability Scale (SUS) and complete a survey on the understandability and perceived usefulness of each tool. Following the completion of both assessments, participants will be asked to nominate their preferred tool and relay their opinions on the tools. The design philosophy of TEMAS and I-MeDeSA differs on the calculation of a score, impacting the way the comparison between the tools can be performed. Convergent validity will be evaluated by matching the items of the two tools with respect to the usability dimensions they assess. SUS scores and answers to the survey will be statistically compared for I-MeDeSA and TEMAS to identify differences. Free-text responses in surveys will be analysed using an inductive approach. ETHICS AND DISSEMINATION Ethical approval is not required in France for a study of this nature. The results will be published in a peer-reviewed journal.
Collapse
Affiliation(s)
- Romaric Marcilly
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France
- Inserm, CIC-IT 1403, F-59000 Lille, France
| | - Wu Yi Zheng
- Black Dog Institute, Randwick, New South Wales, Australia
| | - Regis Beuscart
- Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France
| | - Melissa T Baysari
- The University of Sydney, Faculty of Medicine and Health, School of Medical Sciences, Biomedical Informatics and Digital Health, Sydney, New South Wales, Australia
| |
Collapse
|
3
|
Zheng WY, Van Dort B, Marcilly R, Day R, Burke R, Shakib S, Ku Y, Reid-Anderson H, Baysari M. A Tool for Evaluating Medication Alerting Systems: Development and Initial Assessment. JMIR Med Inform 2021; 9:e24022. [PMID: 34269680 PMCID: PMC8325080 DOI: 10.2196/24022] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2020] [Revised: 11/04/2020] [Accepted: 06/03/2021] [Indexed: 12/31/2022] Open
Abstract
BACKGROUND It is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. Thus, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve the usability of systems. OBJECTIVE This study aims to evaluate a newly developed evidence-based self-assessment tool that allows the valid and reliable evaluation of computerized medication alerting systems. This tool was developed to be used by hospital staff with detailed knowledge of their hospital's computerized provider order entry system and alerts to identify and address potential system deficiencies. In this initial assessment, we aim to determine whether the items in the tool can measure compliance of medication alerting systems with human factors principles of design, the tool can be consistently used by multiple users to assess the same system, and the items are easy to understand and perceived to be useful for assessing medication alerting systems. METHODS The Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consisted of 66 items. In total, 18 staff members recruited across 6 hospitals used the TEMAS to assess their medication alerting systems. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing the results of the TEMAS with those of prior in-house evaluations. Reliability was measured using Krippendorff α to determine agreement among assessors. A 7-item survey was used to determine usability. RESULTS The participants reported mostly negative (n=8) and neutral (n=7) perceptions of alerts in their medication alerting system. However, the validity of the TEMAS could not be directly tested, as participants were unaware of any results from prior in-house evaluations. The reliability of the TEMAS, as measured by Krippendorff α, was low to moderate (range 0.26-0.46); however, participant feedback suggests that individuals' knowledge of the system varied according to their professional background. In terms of usability, 61% (11/18) of participants reported that the TEMAS items were generally easy to understand; however, participants suggested the revision of 22 items to improve clarity. CONCLUSIONS This initial assessment of the TEMAS allowed the identification of its components that required modification to improve usability and usefulness. It also revealed that for the TEMAS to be effective in facilitating a comprehensive assessment of a medication alerting system, it should be completed by a multidisciplinary team of hospital staff from both clinical and technical backgrounds to maximize their knowledge of systems.
Collapse
Affiliation(s)
- Wu Yi Zheng
- Black Dog Institute, Randwick, NSW, Australia.,The University of Sydney, Faculty of Medicine and Health, School of Medical Sciences, Biomedical Informatics and Digital Health, Sydney, Australia
| | - Bethany Van Dort
- The University of Sydney, Faculty of Medicine and Health, School of Medical Sciences, Biomedical Informatics and Digital Health, Sydney, Australia
| | - Romaric Marcilly
- Univ Lille, CHU Lille, ULR 2694, METRICS: Évaluation des Technologies de santé des Pratiques médicales, Lille, France.,INSERM, CHU Lille, CIC-IT/Evalab 1403, Centre d'Investigation Clinique, Lille, France
| | - Richard Day
- University of New South Wales, Randwick, Australia
| | | | | | - Young Ku
- Hunter New England Local Health District, Newcastle, Australia
| | | | - Melissa Baysari
- The University of Sydney, Faculty of Medicine and Health, School of Medical Sciences, Biomedical Informatics and Digital Health, Sydney, Australia
| |
Collapse
|
4
|
Abstract
The implementation of computerised prescribing can result in large reductions in prescribing error rates. The flow-on effects to patient outcomes are not well studied The reduction in errors is dependent on prescribers becoming proficient in using the electronic prescribing system. All potential safety benefits are therefore not expected to be achieved immediately Electronic prescribing systems introduce new types of errors, most frequently errors in selection. Some of these errors can be prevented if the system is well designed Computerised decision support embedded in electronic prescribing systems has enormous potential to improve medication safety. However, current support systems have a limited capacity to provide context-relevant advice to prescribers
Collapse
Affiliation(s)
- Melissa T Baysari
- Faculty of Health Sciences, The University of Sydney.,Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney
| | - Magdalena Z Raban
- Faculty of Health Sciences, The University of Sydney.,Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Sydney
| |
Collapse
|