1
|
Cogan AM, Roberts P, Mallinson T. Using Electronic Health Record Data for Occupational Therapy Health Services Research: Invited Commentary. OTJR-OCCUPATION PARTICIPATION AND HEALTH 2024:15394492241246544. [PMID: 38622903 DOI: 10.1177/15394492241246544] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/17/2024]
Abstract
Health services research (HSR) is a field of study that examines how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affect access to health care, the quality and cost of health care, and health and well-being. HSR approaches can help build the occupational therapy evidence base, particularly in relation to population health. Data from electronic health record (EHR) systems provide a rich resource for applying HSR approaches to examine the value of occupational therapy services. Transparency about data preparation procedures is important for interpreting results. Based on our findings, we describe a six-step cleaning protocol for preparing EHR and billing data from an inpatient rehabilitation facility for research and provide recommendations for the field based on our experience. Using and reporting similar strategies across studies will improve efficiency and transparency, and facilitate comparability of results.
Collapse
Affiliation(s)
| | - Pamela Roberts
- University of Southern California, Los Angeles, USA
- Cedars-Sinai, Los Angeles, CA, USA
- California Rehabilitation Institute, Los Angeles, USA
| | | |
Collapse
|
2
|
Naemi R, Akbarian M, Ebrahimi M, Shahmoradi L, Masoomian B, Rezayi S. Design and evaluation of a web-based electronic health record for amblyopia. Front Med (Lausanne) 2024; 11:1322821. [PMID: 38638930 PMCID: PMC11025453 DOI: 10.3389/fmed.2024.1322821] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Accepted: 03/18/2024] [Indexed: 04/20/2024] Open
Abstract
Introduction Amblyopia, or lazy eye, is a type of visual impairment in which the eyesight is not complete, even with the use of glasses. For the treatment of this disease, accurate and continuous examinations are needed. Nowadays, patient-centered care, by relying on web-based electronic records for amblyopia, has the potential to reduce treatment costs, increase the quality of care, and improve the safety and effectiveness of treatment. Therefore, the purpose of this study is to design and evaluate an Electronic Health Record (EHR) for patients with amblyopia. Methods The present study is applied developmental research. Using a Morgan table as a sampling tool, a straightforward random sampling technique selected 150 records from 1,500 records that were free of flaws. The design of the electronic version proceeded in a cascading manner so that after the design of each part, it was presented to the amblyopia experts, and if approved, the next part was designed. To design this EHR, the C# programming language and MySQL database were used. A system evaluation was performed by entering and recording patient information. For this purpose, the standard Questionnaire of User Interaction Satisfaction (QUIS), consisting of 18 questions, was used. Results According to the amblyopia EHR data elements, the data of physician and patient, examinations, website members, and members' roles were determined. After defining the fields and classes that explain the tables, the EHR was designed. The usability evaluation of the system showed that the mean selection of very good and good options by the users of EHRs was over 90%, indicating the patients' acceptance of web-based EHRs. Conclusion The design of an EHR for amblyopia is an effective step toward integrating and improving the information management of these patients. It will also enable the storage and retrieval of patients' information to reduce and facilitate the control of amblyopia complications.
Collapse
Affiliation(s)
- Roya Naemi
- Department of Health Information Management, School of Paramedical Sciences, Ardabil University of Medical Sciences, Ardabil, Iran
| | - Mina Akbarian
- Department of Health Information Management and Medical Informatics, Tehran University of Medical Sciences, Tehran, Iran
| | - Maryam Ebrahimi
- Department of Health Information Technology, Neyshabur University of Medical Sciences, Neyshabur, Iran
| | - Leila Shahmoradi
- Department of Health Information Management and Medical Informatics, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran
| | - Babak Masoomian
- Farabi Eye Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Sorayya Rezayi
- Department of Health Information Management and Medical Informatics, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran
| |
Collapse
|
3
|
Wohlgemut JM, Pisirir E, Kyrimi E, Stoner RS, Marsh W, Perkins ZB, Tai NRM. Methods used to evaluate usability of mobile clinical decision support systems for healthcare emergencies: a systematic review and qualitative synthesis. JAMIA Open 2023; 6:ooad051. [PMID: 37449057 PMCID: PMC10336299 DOI: 10.1093/jamiaopen/ooad051] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Revised: 06/15/2023] [Accepted: 06/30/2023] [Indexed: 07/18/2023] Open
Abstract
Objective The aim of this study was to determine the methods and metrics used to evaluate the usability of mobile application Clinical Decision Support Systems (CDSSs) used in healthcare emergencies. Secondary aims were to describe the characteristics and usability of evaluated CDSSs. Materials and Methods A systematic literature review was conducted using Pubmed/Medline, Embase, Scopus, and IEEE Xplore databases. Quantitative data were descriptively analyzed, and qualitative data were described and synthesized using inductive thematic analysis. Results Twenty-three studies were included in the analysis. The usability metrics most frequently evaluated were efficiency and usefulness, followed by user errors, satisfaction, learnability, effectiveness, and memorability. Methods used to assess usability included questionnaires in 20 (87%) studies, user trials in 17 (74%), interviews in 6 (26%), and heuristic evaluations in 3 (13%). Most CDSS inputs consisted of manual input (18, 78%) rather than automatic input (2, 9%). Most CDSS outputs comprised a recommendation (18, 78%), with a minority advising a specific treatment (6, 26%), or a score, risk level or likelihood of diagnosis (6, 26%). Interviews and heuristic evaluations identified more usability-related barriers and facilitators to adoption than did questionnaires and user testing studies. Discussion A wide range of metrics and methods are used to evaluate the usability of mobile CDSS in medical emergencies. Input of information into CDSS was predominantly manual, impeding usability. Studies employing both qualitative and quantitative methods to evaluate usability yielded more thorough results. Conclusion When planning CDSS projects, developers should consider multiple methods to comprehensively evaluate usability.
Collapse
Affiliation(s)
- Jared M Wohlgemut
- Corresponding Author: Jared M. Wohlgemut, MSc, Centre for Trauma Sciences, Blizard Institute, Queen Mary University of London, 4 Newark St, London E1 2AT, UK;
| | - Erhan Pisirir
- Department of Electrical Engineering and Computer Science, Queen Mary University of London, London, UK
| | - Evangelia Kyrimi
- Department of Electrical Engineering and Computer Science, Queen Mary University of London, London, UK
| | - Rebecca S Stoner
- Centre for Trauma Sciences, Blizard Institute, Queen Mary University of London, London, UK
- Trauma Service, Royal London Hospital, Barts NHS Health Trust, London, UK
| | - William Marsh
- Department of Electrical Engineering and Computer Science, Queen Mary University of London, London, UK
| | - Zane B Perkins
- Centre for Trauma Sciences, Blizard Institute, Queen Mary University of London, London, UK
- Trauma Service, Royal London Hospital, Barts NHS Health Trust, London, UK
| | - Nigel R M Tai
- Centre for Trauma Sciences, Blizard Institute, Queen Mary University of London, London, UK
- Trauma Service, Royal London Hospital, Barts NHS Health Trust, London, UK
- Academic Department of Military Surgery and Trauma, Royal Centre of Defence Medicine, Birmingham, UK
| |
Collapse
|
4
|
Alves Leite de Barros KA, da Silva Praxedes MF, Pinho Ribeiro AL, Parreiras Martins MA. Effect and usability of mobile health applications for medication adherence in patients with heart failure: A systematic review. Int J Med Inform 2023; 178:105206. [PMID: 37708836 DOI: 10.1016/j.ijmedinf.2023.105206] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Revised: 08/24/2023] [Accepted: 08/29/2023] [Indexed: 09/16/2023]
Abstract
BACKGROUND This review aimed to evaluate the effect of mobile health applications (apps) use on medication adherence in heart failure (HF) patients, and to verify their quality of use. METHODS We searched MEDLINE, Embase, Web of Science, Scopus, CINAHL, LILACS, Cochrane, Clinical Trial Registries, Google Scholar, ProQuest, Open Access Theses and dissertations (OATD), Annual Symposium Proceedings of the American Medical Informatics Association (AMIA). RESULTS Of the 836 records identified, eight studies (five randomized clinical trials (RCTs) and three observational studies) were included, totaling 484 participants, with a mean age ranging from 51.1 to 72 years and 65% were male. The apps improved medication adherence when compared with their comparators in only two RCTs (n = 111, Cohen d = 3.23, 95% confidence interval (CI) 2.66; 3.80 and n = 80, d = 1.20, 95% CI 0.71; 1.66), and two observational studies, one of them with large effect size (n = 142, Cohen d = 1.51, 95% CI 1.12; 1.90). Apps proved to be more effective interventions than their comparators allowed continuous monitoring of patients. All included studies used self-report measures to assess medication adherence behaviors and their results should be interpreted with caution, as such tools may cause social desirability bias or recall bias. CONCLUSION The included studies indicate a possible outcome in favor of the use of apps to improve medication adherence in HF patients, but with very low quality level of evidence. Further studies are needed to investigate the effects of the use of apps, optimal frequency and duration of its use in HF patients.
Collapse
Affiliation(s)
- Kátia Adriana Alves Leite de Barros
- Faculdade de Medicina da Universidade Federal de Minas Gerais, Av. Prof. Alfredo Balena, 190, Bairro Santa Efigênia, Belo Horizonte, Minas Gerais 30130-100, Brazil
| | - Marcus Fernando da Silva Praxedes
- Centro de Ciências da Saúde, Universidade Federal do Recôncavo da Bahia, Av. Carlos Amaral, 1015, Bairro Cajueiro, Santo Antônio de Jesus, Bahia 44430-622, Brazil
| | - Antônio Luiz Pinho Ribeiro
- Faculdade de Medicina da Universidade Federal de Minas Gerais, Av. Prof. Alfredo Balena, 190, Bairro Santa Efigênia, Belo Horizonte, Minas Gerais 30130-100, Brazil; Hospital das Clínicas da Universidade Federal de Minas Gerais, Av. Prof. Alfredo Balena, 110, Bairro Santa Efigênia, Belo Horizonte, Minas Gerais 30130-100, Brazil
| | - Maria Auxiliadora Parreiras Martins
- Faculdade de Medicina da Universidade Federal de Minas Gerais, Av. Prof. Alfredo Balena, 190, Bairro Santa Efigênia, Belo Horizonte, Minas Gerais 30130-100, Brazil; Hospital das Clínicas da Universidade Federal de Minas Gerais, Av. Prof. Alfredo Balena, 110, Bairro Santa Efigênia, Belo Horizonte, Minas Gerais 30130-100, Brazil; Faculdade de Farmácia, Universidade Federal de Minas Gerais, Av. Pres. Antônio Carlos, 6627, Pampulha, Belo Horizonte, Minas Gerais 31270-901, Brazil.
| |
Collapse
|
5
|
Classen DC, Longhurst CA, Davis T, Milstein JA, Bates DW. Inpatient EHR User Experience and Hospital EHR Safety Performance. JAMA Netw Open 2023; 6:e2333152. [PMID: 37695581 PMCID: PMC10495862 DOI: 10.1001/jamanetworkopen.2023.33152] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Accepted: 07/27/2023] [Indexed: 09/12/2023] Open
Abstract
IMPORTANCE Despite the broad adoption and optimization of electronic health record (EHR) systems across the continuum of care, serious usability and safety problems persist. OBJECTIVE To assess whether EHR safety performance is associated with EHR frontline user experience in a national sample of hospitals. DESIGN, SETTING, AND PARTICIPANTS This cross-sectional study included all US adult hospitals that used the National Quality Forum Leapfrog Health IT Safety Measure and also used the ARCH Collaborative EHR User experience survey from January 1, 2017, to January 1, 2019. Data analysis was performed from September 2020 to November 2022. MAIN OUTCOMES AND MEASURES The primary outcomes were hospital performance on the Leapfrog Health IT Safety measure (overall and 10 subcomponents) and the ARCH collaborative frontline user experience scores (overall and 8 subcomponents). Ordinary least squares models with survey responses clustered by hospital were used to assess associations between the overall measures and their subcomponents. RESULTS There were 112 hospitals and 5689 frontline user surveys included in the study. Hospitals scored a mean of 0.673 (range, 0.297-0.973) on the Leapfrog Health IT safety measure; the mean ARCH EHR user experience score was 3.377 (range, 1 [best] to 5 [worst]). The adjusted β coefficient between the overall safety score and overall user experience score was 0.011 (95% CI, 0.006-0.016). The ARCH overall score was also significantly associated with 10 subcategory scores of the Leapfrog Health IT safety score, and the overall Leapfrog score was associated with the 8 subcategory scores of the ARCH user experience score. CONCLUSIONS AND RELEVANCE This cross-sectional study found a positive association between frontline user-rated EHR usability and EHR safety performance. This finding suggests that improving EHR usability, which is a current well-known pain point for EHR users, could have direct benefits in terms of improved EHR safety.
Collapse
Affiliation(s)
- David C. Classen
- Division of Epidemiology, University of Utah School of Medicine, Salt Lake City
- IDEAS Center, VA Salt Lake City Healthcare System, Salt Lake City, Utah
| | - Christopher A. Longhurst
- Department of Medicine, UC San Diego Health, San Diego, California
- Department of Pediatrics, UC San Diego Health, San Diego, California
| | | | - Julia Adler Milstein
- University of California San Francisco Center for Clinical Informatics and Improvement Research, San Francisco
| | - David W. Bates
- Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
- Department of Health Policy and Management, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
| |
Collapse
|
6
|
Kermani F, Kahouei M, Valinejadi A, Sadeghi M, Momeni M, Pahlevanynejad S. Outcome's Classification in Mobile Applications Tailored to Parents of Premature Infants: A Systematic Review. IRANIAN JOURNAL OF PUBLIC HEALTH 2023; 52:1642-1655. [PMID: 37744544 PMCID: PMC10512150 DOI: 10.18502/ijph.v52i8.13402] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/05/2022] [Accepted: 11/20/2022] [Indexed: 09/26/2023]
Abstract
Background Integration of healthcare services for preterm neonates at home and hospital by mobile technology is an economical and convenient intervention, which is being increasingly applied worldwide. We aimed to classify the outcomes of mobile applications tailored to parents of premature infants. Methods This systematic review was conducted by searching the six main databases until May 2021. Mobile applications tailored to parents of premature infants and the reported outcomes of this technology were identified and classified. Quality of screened articles checked by MMAT tool. Results Overall, 10703 articles were retrieved, and after eliminating the duplicated articles, 9 articles were reviewed ultimately. Identified outcomes were categorized into three groups parental, application, and neonatal outcomes. In the parental outcomes, maternal stress/stress coping, parenting self-efficacy, satisfaction, anxiety, partnership advocacy/improved parent-infant relationship, feeling of being safe, reassurance and confidence, increase awareness, as well as discharge preparedness, were identified. In the application outcomes, application usage, ease of use/user-friendly, and usability of the designed application were placed. Finally, the neonatal outcomes include health and clinical items. Conclusion Mobile applications can be useful in prematurity for educating pregnant mothers, managing stress and anxiety, supporting families, and preparing for discharge. Moreover, due to the coronavirus condition, providing remote services for parents is an appropriate solution to reduce the in-person visits to neonatal care centers. Development of tailored apps can promote the neonates' health and reduce their parents' stress.
Collapse
Affiliation(s)
- Farzaneh Kermani
- Health Information Technology Department, School of Allied Medical Sciences, Semnan University of Medical Sciences, Semnan, Iran
| | - Mehdi Kahouei
- Health Information Technology Department, School of Allied Medical Sciences, Semnan University of Medical Sciences, Semnan, Iran
- Social Determinants of Health Research Center, Semnan University of Medical Sciences, Semnan, Iran
| | - Ali Valinejadi
- Health Information Technology Department, School of Allied Medical Sciences, Semnan University of Medical Sciences, Semnan, Iran
- Social Determinants of Health Research Center, Semnan University of Medical Sciences, Semnan, Iran
| | - Malihe Sadeghi
- Health Information Technology Department, School of Allied Medical Sciences, Semnan University of Medical Sciences, Semnan, Iran
| | - Marjan Momeni
- School of Rehabilitation, Semnan University of Medical Sciences, Semnan, Iran
| | - Shahrbanoo Pahlevanynejad
- Health Information Technology Department, School of Allied Medical Sciences, Semnan University of Medical Sciences, Semnan, Iran
| |
Collapse
|
7
|
Alshehri AA, Alanazi A. Usability Study of an Electronic Medical Record From the Nurse Practitioners' Practice: A Qualitative Study Using the Think-Aloud Technique. Cureus 2023; 15:e41603. [PMID: 37565107 PMCID: PMC10411654 DOI: 10.7759/cureus.41603] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/09/2023] [Indexed: 08/12/2023] Open
Abstract
INTRODUCTION Ensuring the usability of electronic medical records (EMRs) is crucial for healthcare providers to offer efficient, effective, and safe patient care. Nurse practitioners (NPs) are integral to the healthcare system and are essential in managing patient workflows. However, few studies assess NPs' perspectives on how EMR usage affects workflow and patient care quality. METHOD In this study, the "think-aloud technique" was utilized for usability testing. It involves observing users (NPs) as they complete their everyday tasks on the EMR while vocalizing their thoughts and emotions. This method has been proven reliable and valid through various research, such as a systematic review. RESULTS The EMR system used by NPs can create a heavy cognitive workload, have limited functionality, can lead to unintended errors, and may experience technical difficulties. CONCLUSION The EMR system used by NPs is challenging due to three main issues: high cognitive workload, limited system functionality, and technical problems. To improve the system, it is recommended to reduce the cognitive burden by customizing the user interface to fit the NPs' needs, enhancing the system's functionality by adding essential features and fixing any technical issues.
Collapse
Affiliation(s)
- Afnan Ali Alshehri
- College of Public Health and Health Informatics, King Saud Bin Abdulaziz University for Health Sciences, Riyadh, SAU
- Research, King Abdullah International Medical Research Center, Riyadh, SAU
| | - Abdullah Alanazi
- College of Public Health and Health Informatics, King Saud Bin Abdulaziz University for Health Sciences, Riyadh, SAU
- Research, King Abdullah International Medical Research Center, Riyadh, SAU
| |
Collapse
|
8
|
Awad S, Amon K, Baillie A, Loveday T, Baysari MT. Human factors and safety analysis methods used in the design and redesign of electronic medication management systems: A systematic review. Int J Med Inform 2023; 172:105017. [PMID: 36809716 DOI: 10.1016/j.ijmedinf.2023.105017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Revised: 01/15/2023] [Accepted: 01/31/2023] [Indexed: 02/11/2023]
Abstract
INTRODUCTION Poorly designed electronic medication management systems (EMMS) or computerized physician order entry (CPOE) systems in hospital settings can result in usability issues and in turn, patient safety risks. As a safety science, human factors and safety analysis methods have potential to support the safe and usable design of EMMS. OBJECTIVE To identify and describe human factors and safety analysis methods that have been used in the design or redesign of EMMS used in hospital settings. MATERIALS AND METHODS A systematic review, following PRISMA guidelines, was conducted by searching online databases and relevant journals from January 2011 to May 2022. Studies were included if they described the practical application of human factors and safety analysis methods to support the design or redesign of a clinician-facing EMMS, or its components. Methods used were extracted and mapped to human centered design (HCD) activities: understanding context of use; specifying user requirements; producing design solutions; and evaluating the design. RESULTS Twenty-one papers met the inclusion criteria. Overall, 21 human factors and safety analysis methods were used in the design or redesign of EMMS with prototyping, usability testing, participant surveys/questionnaires and interviews the most frequent. Human factors and safety analysis methods were most frequently used to evaluate the design of a system (n = 67; 56.3%). Nineteen of 21 (90%) methods used aimed to identify usability issues and/or support iterative design; only one paper utilized a safety-oriented method and one, a mental workload assessment method. DISCUSSION AND CONCLUSION While the review identified 21 methods, EMMS design primarily utilized a subset of available methods, and rarely a method focused on safety. Given the high-risk nature of medication management in complex hospital environments, and the potential for harm due to poorly designed EMMS, there is significant potential to apply more safety-oriented human factors and safety analysis methods to support EMMS design.
Collapse
Affiliation(s)
- Selvana Awad
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Australia; eHealth NSW, Australia.
| | - Krestina Amon
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Australia
| | - Andrew Baillie
- Sydney School of Health Sciences, Faculty of Medicine & Health, The University of Sydney, Australia; Sydney Local Health District, Australia
| | | | - Melissa T Baysari
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Australia
| |
Collapse
|
9
|
Ghorayeb A, Darbyshire JL, Wronikowska MW, Watkinson PJ. Design and validation of a new Healthcare Systems Usability Scale (HSUS) for clinical decision support systems: a mixed-methods approach. BMJ Open 2023; 13:e065323. [PMID: 36717136 PMCID: PMC9887724 DOI: 10.1136/bmjopen-2022-065323] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/31/2023] Open
Abstract
OBJECTIVE To develop and validate a questionnaire to assess the usability of clinical decision support systems (CDSS) and to assist in the early identification of usability issues that may impact patient safety and quality of care. DESIGN Mixed research methods were used to develop and validate the questionnaire. The qualitative study involved scale item development, content and face validity. Pilot testing established construct validity using factor analysis and facilitated estimates for reliability and internal consistency using the Cronbach's alpha coefficient. SETTING Two hospitals within a single National Health Service Trust. PARTICIPANTS We recruited a panel of 7 experts in usability and questionnaire writing for health purposes to test content validity; 10 participants to assess face validity and 78 participants for the pilot testing. To be eligible for this last phase, participants needed to be health professionals with at least 3 months experience using the local hospital electronic patient record system. RESULTS Feedback from the face and content validity phases contributed to the development and improvement of scale items. The final Healthcare Systems Usability Scale (HSUS) proved quick to complete, easy to understand and was mostly worded by potential users. Exploratory analysis revealed four factors related to patient safety, task execution, alerts or recommendations accuracy, the effects of the system on workflow and ease of system use. These separate into four subscales: patient safety and decision effectiveness (seven items), workflow integration (six items), work effectiveness (five items) and user control (four items). These factors affect the quality of care and clinician's ability to make informed and timely decisions when using CDSS. The HSUS has a very good reliability with global Cronbach's alpha 0.914 and between 0.702 and 0.926 for the four subscales. CONCLUSION The HSUS is a valid and reliable tool for usability testing of CDSS and early identification of usability issues that may cause medical adverse events.
Collapse
Affiliation(s)
- Abir Ghorayeb
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Julie L Darbyshire
- Department of Primary Care Health Sciences, Oxford University, Oxford, UK
| | - Marta W Wronikowska
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Peter J Watkinson
- Kadoorie Centre for Critical Care Research and Education, Oxford University Hospitals NHS Trust, Oxford, UK
| |
Collapse
|
10
|
Martins AI, Santinha G, Almeida AM, Ribeiro Ó, Silva T, Rocha N, Silva AG. Consensus on the terms and procedures for planning and reporting usability evaluation of health-related digital solutions: a Delphi study and a resulting checklist (Preprint). J Med Internet Res 2022. [DOI: 10.2196/44326] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/12/2023] Open
|
11
|
Usability of information systems: Experiences of outpatient physicians, outpatient nurses, and open care social welfare professionals from three large cross-sectional surveys in Finland. Int J Med Inform 2022; 165:104836. [DOI: 10.1016/j.ijmedinf.2022.104836] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2022] [Revised: 06/25/2022] [Accepted: 07/19/2022] [Indexed: 11/22/2022]
|
12
|
Buivydaite R, Reen G, Kovalevica T, Dodd H, Hicks I, Vincent C, Maughan D. Improving usability of Electronic Health Records in a UK Mental Health setting: a feasibility study. J Med Syst 2022; 46:50. [PMID: 35674989 PMCID: PMC9177469 DOI: 10.1007/s10916-022-01832-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2020] [Revised: 05/13/2022] [Accepted: 05/16/2022] [Indexed: 11/15/2022]
Abstract
Background Electronic Health Records (EHRs) can help clinicians to plan, document and deliver care for patients in healthcare services. When used consistently, EHRs can advance patient safety and quality, and reduce clinician’s workload. However, usability problems can make it difficult for clinicians to use EHRs effectively, which can negatively impact both healthcare professionals and patients. Objective To improve usability of EHRs within a mental health service in the UK. Methods This was a feasibility study conducted with two mental health teams. A mixed-methods approach was employed. Focus group discussions with clinicians identified existing usability problems in EHRs and changes were made to address these problems. Updated EHR assessment forms were evaluated by comparing the following measures pre and post changes: (1) usability testing to monitor time spent completing and duplicating patient information in EHRs, (2) clinician’s experience of using EHRs, and (3) proportion of completed EHR assessment forms. Results Usability testing with clinicians (n = 3) showed that the time taken to complete EHR assessment forms and time spent duplicating patient information decreased. Clinician’s experience of completing EHR assessment forms also significantly improved post changes compared to baseline (n = 71; p < 0.005). There was a significant increase in completion of most EHR forms by both teams after EHR usability improvements (all at p < 0.01). Conclusions Usability improvements to EHRs can reduce the time taken to complete forms, advance clinician’s experience and increase usage of EHRs. It is important to engage healthcare professionals in the usability improvement process of EHRs in mental health services. Supplementary information The online version contains supplementary material available at 10.1007/s10916-022-01832-0.
Collapse
Affiliation(s)
- Ruta Buivydaite
- Department of Experimental Psychology, University of Oxford, Oxford, UK.
| | - Gurpreet Reen
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | | | - Harry Dodd
- Oxford Health NHS Foundation Trust, Oxford, UK
| | - Ian Hicks
- Oxford Health NHS Foundation Trust, Oxford, UK
| | - Charles Vincent
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | | |
Collapse
|
13
|
Ackermann K, Baker J, Festa M, McMullan B, Westbrook J, Li L. Computerized Clinical Decision Support Systems for the Early Detection of Sepsis Among Pediatric, Neonatal, and Maternal Inpatients: Scoping Review. JMIR Med Inform 2022; 10:e35061. [PMID: 35522467 PMCID: PMC9123549 DOI: 10.2196/35061] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2021] [Revised: 02/27/2022] [Accepted: 03/19/2022] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Sepsis is a severe condition associated with extensive morbidity and mortality worldwide. Pediatric, neonatal, and maternal patients represent a considerable proportion of the sepsis burden. Identifying sepsis cases as early as possible is a key pillar of sepsis management and has prompted the development of sepsis identification rules and algorithms that are embedded in computerized clinical decision support (CCDS) systems. OBJECTIVE This scoping review aimed to systematically describe studies reporting on the use and evaluation of CCDS systems for the early detection of pediatric, neonatal, and maternal inpatients at risk of sepsis. METHODS MEDLINE, Embase, CINAHL, Cochrane, Latin American and Caribbean Health Sciences Literature (LILACS), Scopus, Web of Science, OpenGrey, ClinicalTrials.gov, and ProQuest Dissertations and Theses Global (PQDT) were searched by using a search strategy that incorporated terms for sepsis, clinical decision support, and early detection. Title, abstract, and full-text screening was performed by 2 independent reviewers, who consulted a third reviewer as needed. One reviewer performed data charting with a sample of data. This was checked by a second reviewer and via discussions with the review team, as necessary. RESULTS A total of 33 studies were included in this review-13 (39%) pediatric studies, 18 (55%) neonatal studies, and 2 (6%) maternal studies. All studies were published after 2011, and 27 (82%) were published from 2017 onward. The most common outcome investigated in pediatric studies was the accuracy of sepsis identification (9/13, 69%). Pediatric CCDS systems used different combinations of 18 diverse clinical criteria to detect sepsis across the 13 identified studies. In neonatal studies, 78% (14/18) of the studies investigated the Kaiser Permanente early-onset sepsis risk calculator. All studies investigated sepsis treatment and management outcomes, with 83% (15/18) reporting on antibiotics-related outcomes. Usability and cost-related outcomes were each reported in only 2 (6%) of the 31 pediatric or neonatal studies. Both studies on maternal populations were short abstracts. CONCLUSIONS This review found limited research investigating CCDS systems to support the early detection of sepsis among pediatric, neonatal, and maternal patients, despite the high burden of sepsis in these vulnerable populations. We have highlighted the need for a consensus definition for pediatric and neonatal sepsis and the study of usability and cost-related outcomes as critical areas for future research. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) RR2-10.2196/24899.
Collapse
Affiliation(s)
- Khalia Ackermann
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Australia
| | - Jannah Baker
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Australia
| | - Marino Festa
- Kids Critical Care Research, Department of Paediatric Intensive Care, Children's Hospital at Westmead, Sydney, Australia
| | - Brendan McMullan
- Department of Immunology and Infectious Diseases, Sydney Children's Hospital, Randwick, Sydney, Australia
- Faculty of Medicine & Health, University of New South Wales, Sydney, Australia
| | - Johanna Westbrook
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Australia
| | - Ling Li
- Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Australia
| |
Collapse
|
14
|
Kushniruk A, Banks A, Melton GB, Porta CM, Tignanelli CJ. Barriers to and Facilitators for Acceptance of Comprehensive Clinical Decision Support System-Driven Care Maps for Patients With Thoracic Trauma: Interview Study Among Health Care Providers and Nurses. JMIR Hum Factors 2022; 9:e29019. [PMID: 35293873 PMCID: PMC8968578 DOI: 10.2196/29019] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2021] [Revised: 11/04/2021] [Accepted: 12/19/2021] [Indexed: 01/21/2023] Open
Abstract
BACKGROUND Comprehensive clinical decision support (CDS) care maps can improve the delivery of care and clinical outcomes. However, they are frequently plagued by usability problems and poor user acceptance. OBJECTIVE This study aims to characterize factors influencing successful design and use of comprehensive CDS care maps and identify themes associated with end-user acceptance of a thoracic trauma CDS care map earlier in the process than has traditionally been done. This was a planned adaptive redesign stage of a User Acceptance and System Adaptation Design development and implementation strategy for a CDS care map. This stage was based on a previously developed prototype CDS care map guided by the Unified Theory of Acceptance and Use of Technology. METHODS A total of 22 multidisciplinary end users (physicians, advanced practice providers, and nurses) were identified and recruited using snowball sampling. Qualitative interviews were conducted, audio-recorded, and transcribed verbatim. Generation of prespecified codes and the interview guide was informed by the Unified Theory of Acceptance and Use of Technology constructs and investigative team experience. Interviews were blinded and double-coded. Thematic analysis of interview scripts was conducted and yielded descriptive themes about factors influencing the construction and potential use of an acceptable CDS care map. RESULTS A total of eight dominant themes were identified: alert fatigue (theme 1), automation (theme 2), redundancy (theme 3), minimalistic design (theme 4), evidence based (theme 5), prevent errors (theme 6), comprehensive across the spectrum of disease (theme 7), and malleability (theme 8). Themes 1 to 4 addressed factors directly affecting end users, and themes 5 to 8 addressed factors affecting patient outcomes. More experienced providers prioritized a system that is easy to use. Nurses prioritized a system that incorporated evidence into decision support. Clinicians across specialties, roles, and ages agreed that the amount of extra work generated should be minimal and that the system should help them administer optimal care efficiently. CONCLUSIONS End user feedback reinforces attention toward factors that improve the acceptance and use of a CDS care map for patients with thoracic trauma. Common themes focused on system complexity, the ability of the system to fit different populations and settings, and optimal care provision. Identifying these factors early in the development and implementation process may facilitate user-centered design and improve adoption.
Collapse
Affiliation(s)
| | - Alyssa Banks
- University of Minnesota, Minneapolis, MN, United States
| | - Genevieve B Melton
- Department of Surgery, University of Minnesota, Minneapolis, MN, United States
| | - Carolyn M Porta
- School of Nursing, University of Minnesota, Minneapolis, MN, United States
| | - Christopher J Tignanelli
- Department of Surgery, University of Minnesota, Minneapolis, MN, United States.,Department of Surgery, North Memorial Health Hospital, Robbinsdale, MN, United States
| |
Collapse
|
15
|
Faulkenberry JG, Luberti A, Craig S. Electronic health records, mobile health, and the challenge of improving global health. Curr Probl Pediatr Adolesc Health Care 2022; 52:101111. [PMID: 34969611 DOI: 10.1016/j.cppeds.2021.101111] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
Technology continues to impact healthcare around the world. This provides great opportunities, but also risks. These risks are compounded in low-resource settings where errors in planning and implementation may be more difficult to overcome. Global Health Informatics provides lessons in both opportunities and risks by building off of general Global Health. Global Health Informatics also requires a thorough understanding of the local environment and the needs of low-resource settings. Forming effective partnerships and following the lead of local experts are necessary for sustainability; it also ensures that the priorities of the local community come first. There is an opportunity for partnerships between low-resource settings and high income areas that can provide learning opportunities to avoid the pitfalls that plague many digital health systems and learn how to properly implement technology that truly improves healthcare.
Collapse
Affiliation(s)
- J Grey Faulkenberry
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia.
| | - Anthony Luberti
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia
| | - Sansanee Craig
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia
| |
Collapse
|
16
|
Martikainen S, Salovaara S, Ylönen K, Tynkkynen E, Viitanen J, Tyllinen M, Lääveri T. Social welfare professionals willing to participate in client information system development - Results from a large cross-sectional survey. Inform Health Soc Care 2021; 47:389-402. [PMID: 34877899 DOI: 10.1080/17538157.2021.2010736] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
Human-centered design methods should be implemented throughout the client information system (CIS) development process to understand social welfare professionals' needs, tasks, and contexts of use. The aim of this study was to examine Finnish social welfare professionals' experiences of participating in CIS development.A national cross-sectional web-based survey on the CIS experiences of social welfare professionals (1145 respondents) was conducted in Finland in spring 2019. This study focused on statements concerning the experiences of end users with CIS development and participation. The results are reported by professional and age groups.Half (50%) of the 1145 respondents had participated in CIS development. Half (56%) knew to whom and how to send feedback to software developers, but most (87%) indicated that changes and corrections were not made according to suggestions and quickly enough. The most preferred methods of participation were telling a person in charge of information systems development about usage problems (53%) and showing developers on site how professionals work (34%); 19% were not interested in participating.Social welfare professionals are willing to participate in CIS development, but vendors and social welfare provider organizations are underutilizing this resource. Social welfare informaticists are needed to interpret the needs of end users to software developers.
Collapse
Affiliation(s)
- Susanna Martikainen
- Department of Health and Social Management, University of Eastern Finland, Kuopio, Finland
| | - Samuel Salovaara
- Department of social work, University of Lapland, Rovaniemi, Finland
| | - Katri Ylönen
- Department of Social Sciences and Philosophy, University of Jyväskylä, Jyväskylä, Finland
| | - Elina Tynkkynen
- Department of Information System, University of Jyväskylä, Jyväskylä, Finland
| | - Johanna Viitanen
- Department of Computer Science, Aalto University, Espoo, Finland
| | - Mari Tyllinen
- Department of Computer Science, Aalto University, Espoo, Finland
| | - Tinja Lääveri
- Inflammation Center, Infectious Diseases and Meilahti Vaccine Research Center MeiVac, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
| |
Collapse
|
17
|
Wronikowska MW, Malycha J, Morgan LJ, Westgate V, Petrinic T, Young JD, Watkinson PJ. Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems. J Eval Clin Pract 2021; 27:1403-1416. [PMID: 33982356 PMCID: PMC9438452 DOI: 10.1111/jep.13582] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/18/2020] [Revised: 04/19/2021] [Accepted: 04/20/2021] [Indexed: 01/01/2023]
Abstract
BACKGROUND AND OBJECTIVES Electronic healthcare records have become central to patient care. Evaluation of new systems include a variety of usability evaluation methods or usability metrics (often referred to interchangeably as usability components or usability attributes). This study reviews the breadth of usability evaluation methods, metrics, and associated measurement techniques that have been reported to assess systems designed for hospital staff to assess inpatient clinical condition. METHODS Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology, we searched Medline, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, and Open Grey from 1986 to 2019. For included studies, we recorded usability evaluation methods or usability metrics as appropriate, and any measurement techniques applied to illustrate these. We classified and described all usability evaluation methods, usability metrics, and measurement techniques. Study quality was evaluated using a modified Downs and Black checklist. RESULTS The search identified 1336 studies. After abstract screening, 130 full texts were reviewed. In the 51 included studies 11 distinct usability evaluation methods were identified. Within these usability evaluation methods, seven usability metrics were reported. The most common metrics were ISO9241-11 and Nielsen's components. An additional "usefulness" metric was reported in almost 40% of included studies. We identified 70 measurement techniques used to evaluate systems. Overall study quality was reflected in a mean modified Downs and Black checklist score of 6.8/10 (range 1-9) 33% studies classified as "high-quality" (scoring eight or higher), 51% studies "moderate-quality" (scoring 6-7), and the remaining 16% (scoring below five) were "low-quality." CONCLUSION There is little consistency within the field of electronic health record systems evaluation. This review highlights the variability within usability methods, metrics, and reporting. Standardized processes may improve evaluation and comparison electronic health record systems and improve their development and implementation.
Collapse
Affiliation(s)
| | - James Malycha
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
- Department of Acute Care MedicineUniversity of AdelaideAdelaideAustralia
| | - Lauren J. Morgan
- Nuffield Department of Surgical SciencesUniversity of Oxford, John Radcliffe HospitalOxfordUK
| | - Verity Westgate
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
| | - Tatjana Petrinic
- Bodleian Health Care LibrariesJohn Radcliffe Hospital, University of OxfordOxfordUK
| | - J Duncan Young
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
| | - Peter J. Watkinson
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
| |
Collapse
|
18
|
Iqbal AR, Parau CA, Kazi S, Adams KT, La L, Hettinger AZ, Ratwani RM. Identifying Electronic Medication Administration Record (eMAR) Usability Issues from Patient Safety Event Reports. Jt Comm J Qual Patient Saf 2021; 47:793-801. [PMID: 34657817 DOI: 10.1016/j.jcjq.2021.09.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2021] [Revised: 08/31/2021] [Accepted: 09/08/2021] [Indexed: 10/20/2022]
Abstract
BACKGROUND Improving our understanding of the association between medication errors and health information technology (health IT) usability has the potential to reduce errors and improve patient safety. This study used patient safety event reports (PSEs) to investigate the contribution of usability challenges associated with the electronic medication administration record (eMAR) to medication errors. METHODS Free-text descriptions of 849 medication-related PSEs selected from 2.3 million reports were analyzed. Coders identified the specific health IT components, usability challenge categories, and nuanced usability themes that contributed to each PSE. Thematic analysis was conducted to refine categorizations and identify emerging themes. Final analysis was limited to PSEs involving a contribution from eMAR, either as the point of origin or as a downstream contributor to error. RESULTS eMAR contributed to 473 PSEs. eMAR was the point of origin for 84 (17.8% of 473) PSEs. Usability challenge categories included Workflow support (n = 52, 11.0%) and Display/Visual clutter (n = 30, 6.3%). eMAR contributed downstream from the point of origin in 389 (82.2% of 473) PSEs, with errors stemming primarily from Pharmacy IT and computerized provider order entry (CPOE). Prominent secondary eMAR-associated usability challenges included Display/Visual clutter (n = 327, 69.1%) and Alerting (n = 32, 6.8%). CONCLUSION This study identified several eMAR usability challenges, through the analysis of PSEs, that contribute to medication errors. Findings highlight the critical need for improving the eMAR user interface. Improved interface design, better vendor usability testing, eMAR-focused certification testing, consideration of work system factors, and eMAR-focused usability and safety testing by health care facilities can improve eMAR technology and patient safety.
Collapse
|
19
|
Lloyd S, Long K, Oshni Alvandi A, Di Donato J, Probst Y, Roach J, Bain C. A National Survey of EMR Usability: Comparisons between medical and nursing professions in the hospital and primary care sectors in Australia and Finland. Int J Med Inform 2021; 154:104535. [PMID: 34425552 DOI: 10.1016/j.ijmedinf.2021.104535] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/24/2020] [Revised: 06/09/2021] [Accepted: 06/16/2021] [Indexed: 11/28/2022]
Abstract
BACKGROUND Electronic Medical Record Systems (EMRs) are now part of nursing and medical professionals daily work in the acute and primary care sectors in Australia. Usability is an important factor in their successful adoption and impacts upon clinical workflow, safety and quality, communication, and collaboration. This study replicates a significant body of work conducted by Finnish researchers applying a usability focused survey to understand medical and nursing professionals' experiences in the Australian context. As we implement EMRs across health systems, their usability and design to support clinicians to effectively deliver and document care, is essential. METHODS We conducted an observational study using a cross sectional survey, the National Usability-Focused HIS Scale (NuHISS) developed and validated by Finnish researchers. For this study 13 usability statements collected clinician impressions of EMRs related to technical quality, ease of use, benefits, and collaboration. We report the responses from medical and nursing professionals working in clinical practice settings in Australia, including primary care and hospital sectors in 2020. RESULTS Nursing and medical professionals have different experiences with EMR usability. This depends on the sector they work in and the usability feature measured. In our sample, technical quality features were more positively experienced by doctors in the primary care sector than nurses as well as ease of obtaining patient information and prevention of errors. In the hospital sector nurses experiences with EMRs were more positive with respect to support for routine task completion, learnability, ease of obtaining patient information and entry of patient data. CONCLUSIONS The NuHISS is a suitable tool for measuring the usability experiences of Australian clinicians and the EMRs utilised. Differences in usability experiences were noted between professional groups and sectors. A focus on the usability perspectives of clinicians when enhancing or developing EMR solutions is advocated.
Collapse
Affiliation(s)
| | - Karrie Long
- Nursing Research Hub, The Royal Melbourne Hospital, Melbourne, Australia
| | | | - Josie Di Donato
- QUT Online, Queensland University of Technology, Brisbane, Australia
| | - Yasmine Probst
- School of Medicine, University of Wollongong, Wollongong, NSW, Australia; Illawarra Health and Medical Research Institute, Wollongong, NSW, Australia
| | - Jeremy Roach
- Clinical Informatics Community of Practice, Australasian Institute of Digital Health, Melbourne, Australia
| | - Christopher Bain
- Department of Human Centred Computing, Monash Data Futures Institute, Monash University, Melbourne, Australia
| |
Collapse
|
20
|
Kernebeck S, Busse TS, Jux C, Meyer D, Dreier LA, Zenz D, Zernikow B, Ehlers JP. Participatory Design of an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Study with Nurses and Physicians. CHILDREN-BASEL 2021; 8:children8080695. [PMID: 34438586 PMCID: PMC8392291 DOI: 10.3390/children8080695] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Revised: 08/09/2021] [Accepted: 08/09/2021] [Indexed: 11/30/2022]
Abstract
Background: Electronic medical records (EMRs) offer a promising approach to mapping and documenting the complex information gathered in paediatric palliative care (PPC). However, if they are not well developed, poorly implemented EMRs have unintended consequences that may cause harm to patients. One approach to preventing such harm is the involvement of users in the participatory design to ensure user acceptance and patient safety. Therefore, the aim of this study is to evaluate the acceptance of a novel patient chart module (PCM) as part of an EMR from the perspective of potential users in PPC and to involve these professionals in the design process. Methods: A qualitative observational study with N = 16 PPC professionals (n = 10 nurses, n = 6 physicians) was conducted, including concurrent think aloud (CTA) and semi-structured interviews. A structured content analysis based on the Unified Theory of Acceptance and Use of Technology was applied. Results: The results can be summarized in terms of general observations, performance expectancy, effort expectancy and facilitating conditions, all of which are likely to have a positive influence on acceptance of the PCM from the user perspective in the context of PPC. Conclusions: The involvement of users in the development of EMRs is important for meeting the requirements in PPC. Further software adaptations are necessary to implement these requirements.
Collapse
Affiliation(s)
- Sven Kernebeck
- Chair of Didactics and Educational Research in Health Science, Faculty of Health, Witten/Herdecke University, 58448 Witten, Germany; (T.S.B.); (C.J.); (J.P.E.)
- Correspondence: ; Tel.: +49-(0)2302/926-786-13
| | - Theresa Sophie Busse
- Chair of Didactics and Educational Research in Health Science, Faculty of Health, Witten/Herdecke University, 58448 Witten, Germany; (T.S.B.); (C.J.); (J.P.E.)
| | - Chantal Jux
- Chair of Didactics and Educational Research in Health Science, Faculty of Health, Witten/Herdecke University, 58448 Witten, Germany; (T.S.B.); (C.J.); (J.P.E.)
| | - Dorothee Meyer
- PedScience Research Institute, 45711 Datteln, Germany; (D.M.); (L.A.D.); (B.Z.)
- Department of Children’s Pain Therapy and Pediatric Palliative Care, Faculty of Health, School of Medicine, Witten/Herdecke University, 58448 Witten, Germany
| | - Larissa Alice Dreier
- PedScience Research Institute, 45711 Datteln, Germany; (D.M.); (L.A.D.); (B.Z.)
- Department of Children’s Pain Therapy and Pediatric Palliative Care, Faculty of Health, School of Medicine, Witten/Herdecke University, 58448 Witten, Germany
| | - Daniel Zenz
- Smart-Q Softwaresysteme GmbH, Lise-Meitner-Allee 4, 44801 Bochum, Germany;
| | - Boris Zernikow
- PedScience Research Institute, 45711 Datteln, Germany; (D.M.); (L.A.D.); (B.Z.)
- Department of Children’s Pain Therapy and Pediatric Palliative Care, Faculty of Health, School of Medicine, Witten/Herdecke University, 58448 Witten, Germany
- Pediatric Palliative Care Centre, Children’s and Adolescents’ Hospital, 45711 Datteln, Germany
| | - Jan Peter Ehlers
- Chair of Didactics and Educational Research in Health Science, Faculty of Health, Witten/Herdecke University, 58448 Witten, Germany; (T.S.B.); (C.J.); (J.P.E.)
| |
Collapse
|
21
|
Schreier DJ, Lovely JK. Optimizing Clinical Monitoring Tools to Enhance Patient Review by Pharmacists. Appl Clin Inform 2021; 12:621-628. [PMID: 34161988 DOI: 10.1055/s-0041-1731341] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023] Open
Abstract
BACKGROUND The Clinical Monitoring List (CML) is a real-time scoring system and intervention tool used by Mayo Clinic pharmacists caring for hospitalized patients. OBJECTIVE The study aimed to describe the iterative development and implementation of pharmacist clinical monitoring tools within the electronic health record at a multicampus health system enterprise. METHODS Between October 2018 and January 2019, pharmacists across the enterprise were surveyed to determine opportunities and gaps in CML functionality. Responses were received from 39% (n = 162) of actively staffing inpatient pharmacists. Survey responses identified three main gaps in CML functionality: (1) the desire for automated checklists of tasks, (2) additional rule logic closely aligning with clinical practice guidelines, and (3) the ability to dismiss and defer rules. The failure mode and effect analysis were used to assess risk areas within the CML. To address identified gaps, two A/B testing pilots were undertaken. The first pilot analyzed the effect of updated CML rule logic on pharmacist satisfaction in the domains of automated checklists and guideline alignment. The second pilot assessed the utility of a Clinical Monitoring Navigator (CMN) functioning in conjunction with the CML to display rules with selections to dismiss or defer rules until a user-specified date. The CMN is a workspace to guide clinical end user workflows; permitting the review and actions to be completed within one screen using EHR functionality. RESULTS A total of 27 pharmacists across a broad range of practice specialties were selected for two separate two-week pilot tests. Upon pilot completion, participants were surveyed to assess the effect of updates on performance gaps. CONCLUSION Findings from the enterprise-wide survey and A/B pilot tests were used to inform final build decisions and planned enterprise-wide updated CML and CMN launch. This project serves as an example of the utility of end-user feedback and pilot testing to inform project decisions, optimize usability, and streamline build activities.
Collapse
Affiliation(s)
- Diana J Schreier
- Department of Pharmacy, Mayo Clinic, Rochester, Minnesota, United States
| | - Jenna K Lovely
- Department of Pharmacy, Mayo Clinic, Rochester, Minnesota, United States
| |
Collapse
|
22
|
Wilbanks BA, Moss JA. Impact of Data Entry Interface Design on Cognitive Workload, Documentation Correctness, and Documentation Efficiency. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE PROCEEDINGS. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE 2021; 2021:634-643. [PMID: 34457179 PMCID: PMC8378654] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Clinical documentation serves as the legal record of patient care and used to guide clinical decision making. Inadequately designed data entry user-interfaces may result in unintended consequences that negatively impact patient safety and outcomes because inaccurate information is used to guide clinical decision making. This study utilized an electronic simulated documentation interface (i.e., artificial electronic health record) combined with eye-tracking hardware to analyze documentation correctness, documentation efficiency, and cognitive workload of anesthesia providers (N = 20) generating documentation using different computer-assisted data entry types (drop-down box, radio button, check-box, and free text with autocomplete suggestions). Our study methodology incorporating eye-tracking with electronic health record user interfaces to assess documentation correctness, efficiency, and cognitive workload can be translated to other health care provider types.
Collapse
|
23
|
Tremoulet PD, Shah PD, Acosta AA, Grant CW, Kurtz JT, Mounas P, Kirchhoff M, Wade E. Usability of Electronic Health Record-Generated Discharge Summaries: Heuristic Evaluation. J Med Internet Res 2021; 23:e25657. [PMID: 33856353 PMCID: PMC8085750 DOI: 10.2196/25657] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2020] [Revised: 02/11/2021] [Accepted: 03/16/2021] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Obtaining accurate clinical information about recent acute care visits is extremely important for outpatient providers. However, documents used to communicate this information are often difficult to use. This puts patients at risk of adverse events. Elderly patients who are seen by more providers and have more care transitions are especially vulnerable. OBJECTIVE This study aimed to (1) identify the information about elderly patients' recent acute care visits needed to coordinate their care, (2) use this information to assess discharge summaries, and (3) provide recommendations to help improve the quality of electronic health record (EHR)-generated discharge summaries, thereby increasing patient safety. METHODS A literature review, clinician interviews, and a survey of outpatient providers were used to identify and categorize data needed to coordinate care for recently discharged elderly patients. Based upon those data, 2 guidelines for creating useful discharge summaries were created. The new guidelines, along with 17 previously developed medical documentation usability heuristics, were applied to assess 4 simulated elderly patient discharge summaries. RESULTS The initial research effort yielded a list of 29 items that should always be included in elderly patient discharge summaries and a list of 7 "helpful, but not always necessary" items. Evaluation of 4 deidentified elderly patient discharge summaries revealed that none of the documents contained all 36 necessary items; between 14 and 18 were missing. The documents each had several other issues, and they differed significantly in organization, layout, and formatting. CONCLUSIONS Variations in content and structure of discharge summaries in the United States make them unnecessarily difficult to use. Standardization would benefit both patients, by lowering the risk of care transition-related adverse events, and outpatient providers, by helping reduce frustration that can contribute to burnout. In the short term, acute care providers can help improve the quality of their discharge summaries by working with EHR vendors to follow recommendations based upon this study. Meanwhile, additional human factors work should determine the most effective way to organize and present information in discharge summaries, to facilitate effective standardization.
Collapse
Affiliation(s)
- Patrice D Tremoulet
- Department of Psychology, Rowan University, Glassboro, NJ, United States.,Device Evaluation, ECRI, Plymouth Meeting, PA, United States
| | - Priyanka D Shah
- Device Evaluation, ECRI, Plymouth Meeting, PA, United States
| | - Alisha A Acosta
- Department of Biochemistry, Rowan University, Glassboro, NJ, United States
| | - Christian W Grant
- Department of Psychology, Rowan University, Glassboro, NJ, United States
| | - Jon T Kurtz
- Department of Computer Science, Rowan University, Glassboro, NJ, United States
| | - Peter Mounas
- Department of Biological Sciences, Rowan University, Glassboro, NJ, United States
| | - Michael Kirchhoff
- Department of Emergency Medicine, Cooper Medical School, Rowan University, Camden, NJ, United States
| | | |
Collapse
|
24
|
Silva AG, Caravau H, Martins A, Almeida AMP, Silva T, Ribeiro Ó, Santinha G, Rocha NP. Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults. JMIR Hum Factors 2021; 8:e22774. [PMID: 33439128 PMCID: PMC7840284 DOI: 10.2196/22774] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2020] [Revised: 09/29/2020] [Accepted: 10/29/2020] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND The assessment of usability is a complex process that involves several steps and procedures. It is important to standardize the evaluation and reporting of usability procedures across studies to guide researchers, facilitate comparisons across studies, and promote high-quality usability studies. The first step to standardizing is to have an overview of how usability study procedures are reported across the literature. OBJECTIVE This scoping review of reviews aims to synthesize the procedures reported for the different steps of the process of conducting a user-centered usability assessment of digital solutions relevant for older adults and to identify potential gaps in the present reporting of procedures. The secondary aim is to identify any principles or frameworks guiding this assessment in view of a standardized approach. METHODS This is a scoping review of reviews. A 5-stage scoping review methodology was used to identify and describe relevant literature published between 2009 and 2020 as follows: identify the research question, identify relevant studies, select studies for review, chart data from selected literature, and summarize and report results. The research was conducted on 5 electronic databases: PubMed, ACM Digital Library, IEEE, Scopus, and Web of Science. Reviews that met the inclusion criteria (reporting on user-centered usability evaluation procedures for any digital solution that could be relevant for older adults and were published in English) were identified, and data were extracted for further analysis regarding study evaluators, study participants, methods and techniques, tasks, and test environment. RESULTS A total of 3958 articles were identified. After a detailed screening, 20 reviews matched the eligibility criteria. The characteristics of the study evaluators and participants and task procedures were only briefly and differently reported. The methods and techniques used for the assessment of usability are the topics that were most commonly and comprehensively reported in the reviews, whereas the test environment was seldom and poorly characterized. CONCLUSIONS A lack of a detailed description of several steps of the process of assessing usability and no evidence on good practices of performing it suggests that there is a need for a consensus framework on the assessment of user-centered usability evaluation. Such a consensus would inform researchers and allow standardization of procedures, which are likely to result in improved study quality and reporting, increased sensitivity of the usability assessment, and improved comparability across studies and digital solutions. Our findings also highlight the need to investigate whether different ways of assessing usability are more sensitive than others. These findings need to be considered in light of review limitations.
Collapse
Affiliation(s)
- Anabela G Silva
- School of Health Sciences, Center for Health Technology and Services Research, University of Aveiro, Aveiro, Portugal
| | - Hilma Caravau
- Department of Medical Sciences, Institute of Electronics and Informatics Engineering of Aveiro, Aveiro, Portugal
| | - Ana Martins
- Department of Medical Sciences, Institute of Electronics and Informatics Engineering of Aveiro, Aveiro, Portugal
| | | | - Telmo Silva
- DigiMedia, Department of Communication and Art, University of Aveiro, Aveiro, Portugal
| | - Óscar Ribeiro
- Department of Education and Psychology, Center for Health Technology and Services Research, University of Aveiro, Aveiro, Portugal
| | - Gonçalo Santinha
- Department of Social, Political and Territorial Sciences, Governance, Competitiveness and Public Policies, University of Aveiro, Aveiro, Portugal
| | - Nelson P Rocha
- Department of Medical Sciences, Institute of Electronics and Informatics Engineering of Aveiro, Aveiro, Portugal
| |
Collapse
|
25
|
Melnick ER, Harry E, Sinsky CA, Dyrbye LN, Wang H, Trockel MT, West CP, Shanafelt T. Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis. J Med Internet Res 2020; 22:e23382. [PMID: 33289493 PMCID: PMC7785404 DOI: 10.2196/23382] [Citation(s) in RCA: 22] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2020] [Revised: 10/15/2020] [Accepted: 12/07/2020] [Indexed: 01/22/2023] Open
Abstract
BACKGROUND Electronic health record (EHR) usability and physician task load both contribute to physician professional burnout. The association between perceived EHR usability and workload has not previously been studied at a national level. Better understanding these interactions could give further information as to the drivers of extraneous task load. OBJECTIVE This study aimed to determine the relationship between physician-perceived EHR usability and workload by specialty and evaluate for associations with professional burnout. METHODS A secondary analysis of a cross-sectional survey of US physicians from all specialties was conducted from October 2017 to March 2018. Among the 1250 physicians invited to respond to the subsurvey analyzed here, 848 (67.8%) completed it. EHR usability was assessed with the System Usability Scale (SUS; range: 0-100). Provider task load (PTL) was assessed using the mental demand, physical demand, temporal demand, and effort required subscales of the National Aeronautics and Space Administration Task Load Index (range: 0-400). Burnout was measured using the Maslach Burnout Inventory. RESULTS The mean scores were 46.1 (SD 22.1) for SUS and 262.5 (SD 71.7) for PTL. On multivariable analysis adjusting for age, gender, relationship status, medical specialty, practice setting, hours worked per week, and number of nights on call per week, physician-rated EHR usability was associated with PTL, with each 1-point increase in SUS score (indicating more favorable) associated with a 0.57-point decrease in PTL score (P<.001). On mediation analysis, higher SUS score was associated with lower PTL score, which was associated with lower odds of burnout. CONCLUSIONS A strong association was observed between EHR usability and workload among US physicians, with more favorable usability associated with less workload. Both outcomes were associated with the odds of burnout, with task load acting as a mediator between EHR usability and burnout. Improving EHR usability while decreasing task load has the potential to allow practicing physicians more working memory for medical decision making and patient communication.
Collapse
Affiliation(s)
- Edward R Melnick
- Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States
| | - Elizabeth Harry
- University of Colorado School of Medicine, Aurora, CO, United States
| | - Christine A Sinsky
- Professional Satisfaction and Practice Sustainability, American Medical Association, Chicago, IL, United States
| | - Liselotte N Dyrbye
- Department of Medicine Physician Well-Being Program, Mayo Clinic, Rochester, MN, United States
| | - Hanhan Wang
- Department of Medicine, Stanford School of Medicine, Palo Alto, CA, United States
| | - Mickey Todd Trockel
- Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Colin P West
- Department of Internal Medicine, Mayo Clinic, Rochester, MN, United States
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Tait Shanafelt
- Department of Medicine, Stanford School of Medicine, Palo Alto, CA, United States
| |
Collapse
|
26
|
Zhang T, Booth R, Jean-Louis R, Chan R, Yeung A, Gratzer D, Strudwick G. A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality. JMIR Serious Games 2020; 8:e18153. [PMID: 33112243 PMCID: PMC7657728 DOI: 10.2196/18153] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2020] [Revised: 07/09/2020] [Accepted: 09/03/2020] [Indexed: 11/13/2022] Open
Abstract
Health-related virtual reality (VR) applications for patient treatment, rehabilitation, and medical professional training are on the rise. However, there is little guidance on how to select and perform usability evaluations for VR health interventions compared to the supports that exist for other digital health technologies. The purpose of this viewpoint paper is to present an introductory summary of various usability testing approaches or methods that can be used for VR applications. Along with an overview of each, a list of resources is provided for readers to obtain additionally relevant information. Six categories of VR usability evaluations are described using a previously developed classification taxonomy specific to VR environments: (1) cognitive or task walkthrough, (2) graphical evaluation, (3) post hoc questionnaires or interviews, (4) physical performance evaluation, (5) user interface evaluation, and (6) heuristic evaluation. Given the growth of VR in health care, rigorous evaluation and usability testing is crucial in the development and implementation of novel VR interventions. The approaches outlined in this paper provide a starting point for conducting usability assessments for health-related VR applications; however, there is a need to also move beyond these to adopt those from the gaming industry, where assessments for both usability and user experience are routinely conducted.
Collapse
Affiliation(s)
- Timothy Zhang
- Centre for Addiction and Mental Health, Toronto, ON, Canada
- Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Richard Booth
- Arthur Labatt Family School of Nursing, Western University, London, ON, Canada
| | - Royce Jean-Louis
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| | - Ryan Chan
- Arthur Labatt Family School of Nursing, Western University, London, ON, Canada
| | - Anthony Yeung
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - David Gratzer
- Centre for Addiction and Mental Health, Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Gillian Strudwick
- Centre for Addiction and Mental Health, Toronto, ON, Canada
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
27
|
Logeswaran A, Chong YJ, Edmunds MR. The Electronic Health Record in Ophthalmology: Usability Evaluation Tools for Health Care Professionals. Ophthalmol Ther 2020; 10:13-20. [PMID: 33105019 PMCID: PMC7587158 DOI: 10.1007/s40123-020-00315-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2020] [Accepted: 10/10/2020] [Indexed: 11/24/2022] Open
Abstract
Introduction The adoption of the electronic health record (EHR) has grown rapidly in ophthalmology. However, despite its potential advantages, its implementation has often led to dissatisfaction amongst health care professionals (HCP). This can be addressed using a user centred design (UCD) which is based on the philosophy that ‘the final product should suit the users, rather than making the users suit the product’. There is often no agreed best practice on the role of HCPs in the UCD process. In this paper, we describe practical qualitative methodologies that can be used by HCPs in the design, implementation and evaluation of ophthalmology EHRs. Methods A review of current qualitative usability methodologies was conducted by practising ophthalmologists who are also qualified health informaticians. Results We identified several qualitative methodologies that could be used for EHR evaluation. These include:Tools for user centred design: shadowing and autoethnography, semi-structured interviews and questionnaires Tools for summative testing: card sort and reverse card sort, retrospective think aloud protocol, wireframing, screenshot testing and heat maps
Conclusion High-yield, low-fidelity tools can be used to engage HCPs with the process of ophthalmology EHR design, implementation and evaluation. These methods can be used by HCPs without the requirement for prior training in usability science, and by clinical centres without significant technical requirements.
Collapse
Affiliation(s)
| | - Yu Jeat Chong
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
| | - Matthew R Edmunds
- Department of Ophthalmology, Worcestershire Acute Hospitals NHS Trust, Worcester, UK
| |
Collapse
|
28
|
Senathirajah Y, Kaufman DR, Cato KD, Borycki EM, Fawcett JA, Kushniruk AW. Characterizing and Visualizing Display and Task Fragmentation in the Electronic Health Record: Mixed Methods Design. JMIR Hum Factors 2020; 7:e18484. [PMID: 33084580 PMCID: PMC7641790 DOI: 10.2196/18484] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2020] [Revised: 08/10/2020] [Accepted: 08/21/2020] [Indexed: 01/20/2023] Open
Abstract
BACKGROUND The complexity of health care data and workflow presents challenges to the study of usability in electronic health records (EHRs). Display fragmentation refers to the distribution of relevant data across different screens or otherwise far apart, requiring complex navigation for the user's workflow. Task and information fragmentation also contribute to cognitive burden. OBJECTIVE This study aims to define and analyze some of the main sources of fragmentation in EHR user interfaces (UIs); discuss relevant theoretical, historical, and practical considerations; and use granular microanalytic methods and visualization techniques to help us understand the nature of fragmentation and opportunities for EHR optimization or redesign. METHODS Sunburst visualizations capture the EHR navigation structure, showing levels and sublevels of the navigation tree, allowing calculation of a new measure, the Display Fragmentation Index. Time belt visualizations present the sequences of subtasks and allow calculation of proportion per instance, a measure that quantifies task fragmentation. These measures can be used separately or in conjunction to compare EHRs as well as tasks and subtasks in workflows and identify opportunities for reductions in steps and fragmentation. We present an example use of the methods for comparison of 2 different EHR interfaces (commercial and composable) in which subjects apprehend the same patient case. RESULTS Screen transitions were substantially reduced for the composable interface (from 43 to 14), whereas clicks (including scrolling) remained similar. CONCLUSIONS These methods can aid in our understanding of UI needs under complex conditions and tasks to optimize EHR workflows and redesign.
Collapse
Affiliation(s)
- Yalini Senathirajah
- Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, United States
| | - David R Kaufman
- Medical Informatics Program, School of Health Professions, State University of New York - Downstate Health Sciences University, Brooklyn, NY, United States
| | - Kenrick D Cato
- School of Nursing, Columbia University, New York, NY, United States
| | - Elizabeth M Borycki
- School of Health Information Science, University of Victoria, Victoria, BC, Canada
| | - Jaime Allen Fawcett
- Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, United States
| | - Andre W Kushniruk
- School of Health Information Science, University of Victoria, Victoria, BC, Canada
| |
Collapse
|
29
|
Pierce RP, Eskridge BR, Rehard L, Ross B, Day MA, Belden JL. The Effect of Electronic Health Record Usability Redesign on Annual Screening Rates in an Ambulatory Setting. Appl Clin Inform 2020; 11:580-588. [PMID: 32906152 DOI: 10.1055/s-0040-1715828] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023] Open
Abstract
OBJECTIVES Improving the usability of electronic health records (EHR) continues to be a focus of clinicians, vendors, researchers, and regulatory bodies. To understand the impact of usability redesign of an existing, site-configurable feature, we evaluated the user interface (UI) used to screen for depression, alcohol and drug misuse, fall risk, and the existence of advance directive information in ambulatory settings. METHODS As part of a quality improvement project, based on heuristic analysis, the existing UI was redesigned. Using an iterative, user-centered design process, several usability defects were corrected. Summative usability testing was performed as part of the product development and implementation cycle. Clinical quality measures reflecting rolling 12-month rates of screening were examined over 8 months prior to the implementation of the redesigned UI and 9 months after implementation. RESULTS Summative usability testing demonstrated improvements in task time, error rates, and System Usability Scale scores. Interrupted time series analysis demonstrated significant improvements in all screening rates after implementation of the redesigned UI compared with the original implementation. CONCLUSION User-centered redesign of an existing site-specific UI may lead to significant improvements in measures of usability and quality of patient care.
Collapse
Affiliation(s)
- Robert P Pierce
- Department of Family and Community Medicine, University of Missouri, Columbia, Missouri, United States
| | - Bernie R Eskridge
- Department of Child Health, University of Missouri, Columbia, Missouri, United States
| | - LeAnn Rehard
- Nursing Informatics, University of Missouri Health Care, Columbia, Missouri, United States
| | - Brandi Ross
- Tiger Institute, Cerner Corporation, Columbia, Missouri, United States
| | - Margaret A Day
- Department of Family and Community Medicine, University of Missouri, Columbia, Missouri, United States
| | - Jeffery L Belden
- Department of Family and Community Medicine, University of Missouri, Columbia, Missouri, United States.,Tiger Institute, Cerner Corporation, Columbia, Missouri, United States
| |
Collapse
|
30
|
Martikainen S, Kaipio J, Lääveri T. End-user participation in health information systems (HIS) development: Physicians' and nurses’ experiences. Int J Med Inform 2020; 137:104117. [DOI: 10.1016/j.ijmedinf.2020.104117] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2019] [Revised: 02/12/2020] [Accepted: 03/03/2020] [Indexed: 11/30/2022]
|
31
|
Kaipio J, Kuusisto A, Hyppönen H, Heponiemi T, Lääveri T. Physicians’ and nurses’ experiences on EHR usability: Comparison between the professional groups by employment sector and system brand. Int J Med Inform 2020; 134:104018. [DOI: 10.1016/j.ijmedinf.2019.104018] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2019] [Revised: 08/30/2019] [Accepted: 10/18/2019] [Indexed: 12/31/2022]
|
32
|
Almeida AF, Rocha NP, Silva AG. Methodological Quality of Manuscripts Reporting on the Usability of Mobile Applications for Pain Assessment and Management: A Systematic Review. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2020; 17:E785. [PMID: 32012674 PMCID: PMC7038093 DOI: 10.3390/ijerph17030785] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/20/2019] [Revised: 01/16/2020] [Accepted: 01/24/2020] [Indexed: 01/10/2023]
Abstract
BACKGROUND There has been increasing use of mobile mHealth applications, including pain assessment and pain self-management apps. The usability of mHealth applications has vital importance as it affects the quality of apps. Thus, usability assessment with methodological rigor is essential to minimize errors and undesirable consequences, as well as to increase user acceptance. OBJECTIVE this study aimed to synthesize and evaluate existing studies on the assessment of the usability of pain-related apps using a newly developed scale. METHODS an electronic search was conducted in several databases, combining relevant keywords. Then titles and abstracts were screened against inclusion and exclusion criteria. The eligible studies were retrieved and independently screened for inclusion by two authors. Disagreements were resolved by discussion until consensus was reached. RESULTS a total of 31 articles were eligible for inclusion. Quality assessment revealed that most manuscripts did not assess usability using valid instruments or triangulation of methods of usability assessment. Most manuscripts also failed to assess the three domains of usability (effectiveness, efficiency and satisfaction). CONCLUSIONS future studies should consider existing guidelines on usability assessment design, development and assessment of pain-related apps.
Collapse
Affiliation(s)
- Ana F. Almeida
- Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Campus Universitário de Santiago, 3810-193 Aveiro, Portugal (N.P.R.)
| | - Nelson P. Rocha
- Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Campus Universitário de Santiago, 3810-193 Aveiro, Portugal (N.P.R.)
- Department of Medical Sciences, Universidade de Aveiro - Edifício 30, Agras do Crasto - Campus Universitário de Santiago, 3810-193 Aveiro, Portugal
| | - Anabela G. Silva
- School of Health Sciences, Universidade de Aveiro - Edifício 30, Agras do Crasto - Campus Universitário de Santiago, 3810-193 Aveiro, Portugal
- Center for Health Technology and Services Research, Universidade de Aveiro (CINTESIS.UA), Campus Universitário de Santiago, 3810-193 Aveiro, Portugal
| |
Collapse
|
33
|
Schopf TR, Nedrebø B, Hufthammer KO, Daphu IK, Lærum H. How well is the electronic health record supporting the clinical tasks of hospital physicians? A survey of physicians at three Norwegian hospitals. BMC Health Serv Res 2019; 19:934. [PMID: 31801518 PMCID: PMC6894258 DOI: 10.1186/s12913-019-4763-0] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2019] [Accepted: 11/20/2019] [Indexed: 01/20/2023] Open
Abstract
Background The electronic health record is expected to improve the quality and efficiency of health care. Many novel functionalities have been introduced in order to improve medical decision making and communication between health care personnel. There is however limited evidence on whether these new functionalities are useful. The aim of our study was to investigate how well the electronic health record system supports physicians in performing basic clinical tasks. Methods Physicians of three prominent Norwegian hospitals participated in the survey. They were asked, in an online questionnaire, how well the hospital’s electronic health record system DIPS supported 49 clinical tasks as well as how satisfied they were with the system in general, including the technical performance. Two hundred and eight of 402 physicians (52%) submitted a completely answered questionnaire. Results Seventy-two percent of the physicians had their work interrupted or delayed because the electronic health record hangs or crashes at least once a week, while 22% had experienced this problem daily. Fifty-three percent of the physicians indicated that the electronic health record is cumbersome to use and adds to their workload. The majority of physicians were satisfied with managing tests, e.g., requesting laboratory tests, reading test results and managing radiological investigations and electrocardiograms. Physicians were less satisfied with managing referrals. There was high satisfaction with some of the decision support functionalities available for prescribing drugs. This includes drug interaction alerts and drug allergy warnings, which are displayed automatically. However, physicians were less satisfied with other aspects of prescribing drugs, including getting an overview of the ongoing drug therapy. Conclusions In the survey physicians asked for improvements of certain electronic health record functionalities like medication, clinical workflow support including planning and better overviews. In addition, there is apparently a need to focus on system stability, number of logins, reliability and better instructions on available electronic health record features. Considerable development is needed in current electronic health record systems to improve usefulness and satisfaction.
Collapse
Affiliation(s)
- Thomas Roger Schopf
- Norwegian Centre for E-health Research, University Hospital of North-Norway, P.O. Box 35, 9038, Tromsø, Norway.
| | - Bente Nedrebø
- Norwegian Board of Health Supervision, P.O. Box 231 Skøyen, 0213, Oslo, Norway
| | - Karl Ove Hufthammer
- Centre for Clinical Research, Haukeland University Hospital, P.O. Box 1400, 5021, Bergen, Norway
| | - Inderjit Kaur Daphu
- Section for e-Health, Department for Research and Development, Haukeland University Hospital, P.O. Box 1400, 5021, Bergen, Norway
| | - Hallvard Lærum
- The Norwegian Directorate for e-health, P.O. Box 221 Skøyen, 0213, Oslo, Norway
| |
Collapse
|
34
|
Silva AG, Simões P, Santos R, Queirós A, Rocha NP, Rodrigues M. A Scale to Assess the Methodological Quality of Studies Assessing Usability of Electronic Health Products and Services: Delphi Study Followed by Validity and Reliability Testing. J Med Internet Res 2019; 21:e14829. [PMID: 31730036 PMCID: PMC6884719 DOI: 10.2196/14829] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2019] [Revised: 07/26/2019] [Accepted: 08/13/2019] [Indexed: 12/27/2022] Open
Abstract
BACKGROUND The usability of electronic health (eHealth) and mobile health apps is of paramount importance as it impacts the quality of care. Methodological quality assessment is a common practice in the field of health for different designs and types of studies. However, we were unable to find a scale to assess the methodological quality of studies on the usability of eHealth products or services. OBJECTIVE This study aimed to develop a scale to assess the methodological quality of studies assessing usability of mobile apps and to perform a preliminary analysis of of the scale's feasibility, reliability, and construct validity on studies assessing usability of mobile apps, measuring aspects of physical activity. METHODS A 3-round Delphi panel was used to generate a pool of items considered important when assessing the quality of studies on the usability of mobile apps. These items were used to write the scale and the guide to assist its use. The scale was then used to assess the quality of studies on usability of mobile apps for physical activity, and it assessed in terms of feasibility, interrater reliability, and construct validity. RESULTS A total of 25 experts participated in the Delphi panel, and a 15-item scale was developed. This scale was shown to be feasible (time of application mean 13.10 [SD 2.59] min), reliable (intraclass correlation coefficient=0.81; 95% CI 0.55-0.93), and able to discriminate between low- and high-quality studies (high quality: mean 9.22 [SD 0.36]; low quality: mean 6.86 [SD 0.80]; P=.01). CONCLUSIONS The scale that was developed can be used both to assess the methodological quality of usability studies and to inform its planning.
Collapse
Affiliation(s)
- Anabela G Silva
- School of Health Sciences, University of Aveiro, Aveiro, Portugal
| | - Patrícia Simões
- School of Health Sciences, University of Aveiro, Aveiro, Portugal
| | - Rita Santos
- Higher School of Technology and Management of Águeda, Aveiro, Portugal
| | | | - Nelson P Rocha
- Department of Medical Sciences, University of Aveiro, Aveiro, Portugal
| | - Mário Rodrigues
- Higher School of Technology and Management of Águeda, Aveiro, Portugal
| |
Collapse
|
35
|
Gizaw T, Bogale M, Alemayehu T. Evaluation of the electronic health record system in maternal and child health centers of Marie Stopes International Ethiopia. Gates Open Res 2019. [DOI: 10.12688/gatesopenres.13001.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
Background: The Health Information Management Systems Society (HIMSS) defines an Electronic Health Record (EHR) as a longitudinal electronic record of patient health information generated by one or more encounters in any care delivery setting”. EHR systems improve quality and adaptability of care, contributing to patient safety. The system is also efficient and provides real-time administration to patients. Driven by the need to facilitate clinical and administrative process, and to reduce medical errors, Marie Stopes international Ethiopia (MSIE) implemented EHR system in its maternal and child health (MCH) centers. This study sought to evaluate the EHR system in six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers) of these MCH centers in terms of ease of performing tasks and satisfaction with the system. Methods: A cross-sectional method with formative evaluation was used to address the objective. A sample size of 54 were used for the investigation. The participants were from six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers) within three MCH centers. Data was collected using structured, self-administered questionnaires and interviews and analyzed using SPSS 20. Results: In total, 83% of respondents found that performing tasks using EHR made tasks easier; the lowest score was from Pharmacy department (48%). The overall satisfaction with EHR was 87%. The lowest score is from the indicator “the system meets my needs” (80%). Conclusions: Staffs in all department reported the EHR system as “very good” for both ease of performing their daily tasks and satisfaction with the system. In addition, the study revealed that ease of performing tasks and satisfaction with the system were varied by department and specific task within the department. It is essential to know the needs and requirements of each department before implementation of the system and user feedback for long lasting uptake and impact.
Collapse
|
36
|
Wright MC, Borbolla D, Waller RG, Del Fiol G, Reese T, Nesbitt P, Segall N. Critical care information display approaches and design frameworks: A systematic review and meta-analysis. J Biomed Inform 2019; 3:100041. [PMID: 31423485 PMCID: PMC6696941 DOI: 10.1016/j.yjbinx.2019.100041] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2018] [Revised: 06/10/2019] [Accepted: 06/16/2019] [Indexed: 11/06/2022]
Abstract
OBJECTIVE To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians. METHODS We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author. RESULTS Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible. CONCLUSION Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.
Collapse
Affiliation(s)
- Melanie C. Wright
- Trinity Health, Livonia, MI, USA
- Saint Alphonsus Regional Medical Center, Boise, ID, USA
| | - Damian Borbolla
- Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | | | | | - Thomas Reese
- Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - Paige Nesbitt
- Saint Alphonsus Regional Medical Center, Boise, ID, USA
| | - Noa Segall
- Anesthesiology, Duke University, Durham, NC, USA
| |
Collapse
|
37
|
Jacobs M, Boersma LJ, Swart R, Mannens R, Reymen B, Körver F, van Merode F, Dekker A. Electronic Health Record implementation in a large academic radiotherapy department: Temporarily disruptions but long-term benefits. Int J Med Inform 2019; 129:342-348. [DOI: 10.1016/j.ijmedinf.2019.07.008] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2018] [Revised: 03/22/2019] [Accepted: 07/09/2019] [Indexed: 10/26/2022]
|
38
|
Brown N, Eghdam A, Koch S. Usability Evaluation of Visual Representation Formats for Emergency Department Records. Appl Clin Inform 2019; 10:454-470. [PMID: 31242513 PMCID: PMC6594835 DOI: 10.1055/s-0039-1692400] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022] Open
Abstract
BACKGROUND Integration of electronic information is a challenge for multitasking emergency providers, with implications for patient safety. Visual representations can assist sense-making of complex data sets; however, benefit and acceptability in emergency care is unproven. OBJECTIVES This article evaluates visually focused alternatives to lists or tabular formats, to better understand possible usability in Emergency Department Information System (EDIS). METHODS A counterbalanced, repeated-measures experiment, satisfaction surveys, and narrative content analysis was conducted remotely by Web platform. Participants were 37 American emergency physicians; they completed 16 clinical cases comparing 4 visual designs to the control formats from a commercially available EDIS. They then evaluated two additional chart overview representations without controls. RESULTS Visual designs provided benefit in several areas compared to controls. Task correctness (90% to 76%; p = 0.003) and completion time (median: 49-74 seconds; p < 0.001) were superior for a medication history timeline with class and schedule highlighting. Completion time (median: 45-60 seconds; p = 0.03) was superior for a past medical history design, using pertinent diagnosis codes in highlighting rules. Less mental effort was reported for visual allergy (p = 0.04), past medical history (p < 0.001), and medication timeline (p < 0.001) designs. Most of the participants agreed with statements of likeability, preference, and benefit for visual designs; nonetheless, contrary opinions were seen, and more complex designs were viewed less favorably. CONCLUSION Physician performance with visual representations of clinical data can in some cases exceed standard formats, even in absence of training. Highlighting of priority clinical categories was rated easier-to-use on average than unhighlighted controls. Perceived complexity of timeline representations can limit desirability for a subset of users, despite potential benefit.
Collapse
Affiliation(s)
- Nathaniel Brown
- Department of Learning, Informatics, Management and Ethics, Health Informatics Centre, Karolinska Institutet, Stockholm, Sweden.,Department of Emergency Medicine, Medical College of Wisconsin, Milwaukee, Wisconsin, United States
| | - Aboozar Eghdam
- Department of Learning, Informatics, Management and Ethics, Health Informatics Centre, Karolinska Institutet, Stockholm, Sweden
| | - Sabine Koch
- Department of Learning, Informatics, Management and Ethics, Health Informatics Centre, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
39
|
Ratwani RM, Savage E, Will A, Fong A, Karavite D, Muthu N, Rivera AJ, Gibson C, Asmonga D, Moscovitch B, Grundmeier R, Rising J. Identifying Electronic Health Record Usability And Safety Challenges In Pediatric Settings. Health Aff (Millwood) 2019; 37:1752-1759. [PMID: 30395517 DOI: 10.1377/hlthaff.2018.0699] [Citation(s) in RCA: 84] [Impact Index Per Article: 16.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
Pediatric populations are uniquely vulnerable to the usability and safety challenges of electronic health records (EHRs), particularly those related to medication, yet little is known about the specific issues contributing to hazards. To understand specific usability issues and medication errors in the care of children, we analyzed 9,000 patient safety reports, made in the period 2012-17, from three different health care institutions that were likely related to EHR use. Of the 9,000 reports, 3,243 (36 percent) had a usability issue that contributed to the medication event, and 609 (18.8 percent) of the 3,243 might have resulted in patient harm. The general pattern of usability challenges and medication errors were the same across the three sites. The most common usability challenges were associated with system feedback and the visual display. The most common medication error was improper dosing.
Collapse
Affiliation(s)
- Raj M Ratwani
- Raj M. Ratwani ( ) is director of the National Center for Human Factors in Healthcare, MedStar Health, and an assistant professor of emergency medicine, Department of Emergency Medicine, Georgetown University School of Medicine, both in Washington, D.C
| | - Erica Savage
- Erica Savage is a manager in Ambulatory Quality and Safety, MedStar Health
| | - Amy Will
- Amy Will is a research program manager at the National Center for Human Factors in Healthcare, MedStar Health
| | - Allan Fong
- Allan Fong is a research scientist at the National Center for Human Factors in Healthcare, MedStar Health
| | - Dean Karavite
- Dean Karavite is principal human computer interaction specialist, Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, in Pennsylvania
| | - Naveen Muthu
- Naveen Muthu is director of the Cognitive Informatics Group, Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, and an instructor of pediatrics, University of Pennsylvania Perelman School of Medicine
| | - A Joy Rivera
- A. Joy Rivera is a senior human factors system engineer at the Children's Hospital of Wisconsin, in Milwaukee
| | - Cori Gibson
- Cori Gibson is a safety specialist at the Children's Hospital of Wisconsin
| | - Don Asmonga
- Don Asmonga is an officer in the Health Information Technology Initiative, Pew Charitable Trusts, in Washington, D.C
| | - Ben Moscovitch
- Ben Moscovitch is the project director of the Health Information Technology Initiative, Pew Charitable Trusts
| | - Robert Grundmeier
- Robert Grundmeier is director of clinical informatics, Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, and an assistant professor of pediatrics, University of Pennsylvania Perelman School of Medicine
| | - Josh Rising
- Josh Rising is director of Healthcare Programs, Pew Health Group, Pew Charitable Trusts
| |
Collapse
|
40
|
Sutton JM, Ash SR, Al Makki A, Kalakeche R. A Daily Hospital Progress Note that Increases Physician Usability of the Electronic Health Record by Facilitating a Problem-Oriented Approach to the Patient and Reducing Physician Clerical Burden. Perm J 2019; 23:18-221. [PMID: 31314721 DOI: 10.7812/tpp/18-221] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
We suggest changes in the electronic health record (EHR) in hospitalized patients to increase EHR usability by optimizing the physician's ability to approach the patient in a problem-oriented fashion and by reducing physician data entry and chart navigation. The framework for these changes is a Physician's Daily Hospital Progress Note organized into 3 sections: Subjective, Objective, and a combined Assessment and Plan section, subdivided by problem titles. The EHR would consolidate information for each problem by: 1) juxtaposing to each problem title relevant medications, key durable results, and limitations; 2) entering in the running lists under Assessment and Plan the most relevant information for that day, including abbreviated versions of relevant reports; and 3) generating a flow sheet in a problem's progress note for any key results tracked daily. To reduce physician EHR navigation, the EHR would place in the Objective section abbreviated versions of notes of other physicians, nurses, and allied health professionals as well as recent orders. The physician would enter only the analysis and plan and new information not included in the EHR. The consolidation of information for each problem would facilitate physician communication at points of transition of care including generation of a problem-oriented discharge summary.
Collapse
Affiliation(s)
- James M Sutton
- Department of Nephrology, Indiana University Health, Lafayette
| | - Steven R Ash
- Department of Nephrology, Indiana University Health, Lafayette
| | - Akram Al Makki
- Department of Nephrology, Indiana University Health, Lafayette
| | - Rabih Kalakeche
- Department of Nephrology, Indiana University Health, Lafayette
| |
Collapse
|
41
|
Kennedy B, Kerns E, Chan YR, Chaparro BS, Fouquet SD. Safeuristics! Do Heuristic Evaluation Violation Severity Ratings Correlate with Patient Safety Severity Ratings for a Native Electronic Health Record Mobile Application? Appl Clin Inform 2019; 10:210-218. [PMID: 30919397 DOI: 10.1055/s-0039-1681073] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022] Open
Abstract
OBJECTIVE Usability of electronic health records (EHRs) remains challenging, and poor EHR design has patient safety implications. Heuristic evaluation detects usability issues that can be classified by severity. The National Institute of Standards and Technology provides a safety scale for EHR usability. Our objectives were to investigate the relationship between heuristic severity ratings and safety scale ratings in an effort to analyze EHR safety. MATERIALS AND METHODS Heuristic evaluation was conducted on seven common mobile EHR tasks, revealing 58 heuristic violations and 28 unique usability issues. Each usability issue was independently scored for severity by trained hospitalists and a Human Factors researcher and for safety severity by two physician informaticists and two clinical safety professionals. RESULTS Results demonstrated a positive correlation between heuristic severity and safety severity ratings. Regression analysis demonstrated that 49% of safety risk variability by clinical safety professionals (r = 0.70; n = 28) and 42% of safety risk variability by clinical informatics specialists (r = 0.65; n = 28) was explained by usability severity scoring of problems outlined by heuristic evaluation. Higher severity ratings of the usability issues were associated with increased perceptions of patient safety risk. DISCUSSION This study demonstrated the use of heuristic evaluation as a technique to quickly identify usability problems in an EHR that could lead to safety issues. Detection of higher severity ratings could help prioritize failures in EHR design that more urgently require design changes. This approach is a cost-effective technique for improving usability while impacting patient safety. CONCLUSION Results from this study demonstrate the efficacy of the heuristic evaluation technique to identify usability problems that impact safety of the EHR. Also, the use of interdisciplinary teams for evaluation should be considered for severity assessment.
Collapse
Affiliation(s)
- Brandan Kennedy
- Medical Informatics and Telemedicine, Children's Mercy Hospital, Kansas City, Missouri, United States.,Pediatric Hospital Medicine, Children's Mercy Hospital, Kansas City, Missouri, United States.,Human Factors Collaborative, Children's Mercy Hospital, Kansas City, Missouri, United States
| | - Ellen Kerns
- Department of Pediatrics, University of Nebraska Medical Center, Omaha, Nebraska, United States
| | - Y Raymond Chan
- Pediatric Hospital Medicine, Children's Mercy Hospital, Kansas City, Missouri, United States.,Human Factors Collaborative, Children's Mercy Hospital, Kansas City, Missouri, United States
| | - Barbara S Chaparro
- Human Factors and Behavioral Neurobiology, Embry-Riddle Aeronautical University, Daytona Beach, Florida, United States
| | - Sarah D Fouquet
- Human Factors Collaborative, Children's Mercy Hospital, Kansas City, Missouri, United States
| |
Collapse
|
42
|
Georgsson M, Staggers N, Årsand E, Kushniruk A. Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: A case study and beginning method validation. J Biomed Inform 2019; 91:103110. [PMID: 30721757 DOI: 10.1016/j.jbi.2019.103110] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2018] [Revised: 12/22/2018] [Accepted: 01/17/2019] [Indexed: 01/07/2023]
Abstract
INTRODUCTION Self-management of chronic diseases using mobile health (mHealth) systems and applications is becoming common. Current evaluation methods such as formal usability testing can be very costly and time-consuming; others may be more efficient but lack a user focus. We propose an enhanced cognitive walkthrough (CW) method, the user-centered CW (UC-CW), to address identified deficiencies in the original technique and perform a beginning validation with think aloud protocol (TA) to assess its effectiveness, efficiency and user acceptance in a case study with diabetes patient users on a mHealth self-management application. MATERIALS AND METHODS A total of 12 diabetes patients at University of Utah Health, USA, were divided into UC-CW and think aloud (TA) groups. The UC-CW method included: making the user the main evaluator for detecting usability problems, having a dual domain facilitator, and using three other improved processes: validated task development, higher level tasks and a streamlined evaluation process. Users interacted with the same mHealth application for both methods. Post-evaluation assessments included the NASA RTLX instrument and a set of brief interview questions. RESULTS Participants had similar demographic characteristics. A total of 26 usability problems were identified with the UC-CW and 20 with TA. Both methods produced similar ratings: severity across all views (UC-CW = 2.7 and TA = 2.6), numbers of problems in the same views (Main View [UC-CW = 11, TA = 10], Carbohydrate Entry View [UC-CW = 4, TA = 3] and List View [UC-CW = 3, TA = 3]) with similar heuristic violations (Match Between the System and Real World [UC-CW = 19, TA = 16], Consistency and Standards [UC-CW = 17, TA = 15], and Recognition Rather than Recall [UC-CW = 13, TA = 10]). Both methods converged on eight usability problems, but the UC-CW group detected five critical issues while the TA group identified two. The UC-CW group identified needed personalized features for patients' disease needs not identified with TA. UC-CW was more efficient on average time per identified usability problem and on the total evaluation process with patients. NASA RTLX scores indicated that participants experienced the UC-CW half as cognitively demanding. Common themes from interviews indicated the UC-CW as enjoyable and easy to perform while TA was considered somewhat awkward and more cognitively challenging. CONCLUSIONS UC-CW was effective for finding severe, recurring usability problems and it highlighted the need for personalized user features. The method was also efficient and had high user acceptance. These results indicate UC-CW's utility and user acceptance in evaluating a mHealth self-management application. It provides an additional usability evaluation technique for researchers.
Collapse
Affiliation(s)
- Mattias Georgsson
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA; Faculty of Computing, Blekinge Institute of Technology, Karlskrona, Sweden; Department of Health Sciences, University West, Trollhättan, Sweden
| | - Nancy Staggers
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA; Summit Health Informatics, Salt Lake City, UT, USA
| | - Eirik Årsand
- The Norwegian Centre for E-health Research, University Hospital of North Norway, Tromsø, Norway; UiT The Arctic University of Norway, Department of Clinical Medicine, Tromsø, Norway
| | - Andre Kushniruk
- School of Health Information Science, University of Victoria, Victoria, Canada
| |
Collapse
|
43
|
Liu VX, Haq N, Chan IC, Hoberman B. Inpatient electronic health record maintenance from 2010 to 2015. THE AMERICAN JOURNAL OF MANAGED CARE 2019; 25:18-21. [PMID: 30667607 PMCID: PMC6596284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
OBJECTIVES To describe the scale and scope of inpatient electronic health record (EHR) maintenance following initial implementation. STUDY DESIGN A retrospective study reviewing EHR change documentation within an integrated healthcare delivery system that has 21 hospitals. METHODS Between 2010 and 2015, we identified and categorized all significant changes made to the inpatient EHR, as documented within monthly EHR communication updates. We categorized EHR changes as updates to existing functionality or upgrades to new functionality. We grouped changes within larger functional domains as orders, alerts and customization, surgical and emergency department (ED), data review, reports and health information management, and other. We also identified the clinical areas and user roles targeted by these changes. RESULTS Over a 6-year period, 5551 unique changes were made to the inpatient EHR, with a median of 72 changes per month. Changes most frequently targeted orders (44.7% of 2190 change documents) and order sets (29.9% of documents). In total, changes affected 135 EHR functions. A total of 151 unique user roles were affected by these changes, with the most frequent roles including nurses (30.6%), physicians (26.6%), and other clinical staff (22.7%). The clinical areas most targeted by changes included surgical areas and the ED. CONCLUSIONS Over 6 years, EHR maintenance for clinical functionality was substantial and varied with pervasive impacts, requiring persistent attention, diverse expertise, and interdisciplinary collaboration.
Collapse
Affiliation(s)
- Vincent X Liu
- Kaiser Permanente Division of Research, 2000 Broadway, Oakland, CA 95070.
| | | | | | | |
Collapse
|
44
|
Lee Y, Jung MY, Shin GW, Bahn S, Park T, Cho I, Lee JH. Safety and Usability Guidelines of Clinical Information Systems Integrating Clinical Workflow: A Systematic Review. Healthc Inform Res 2018; 24:157-169. [PMID: 30109149 PMCID: PMC6085203 DOI: 10.4258/hir.2018.24.3.157] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2018] [Revised: 07/08/2018] [Accepted: 07/12/2018] [Indexed: 11/23/2022] Open
Abstract
Objectives The usability of clinical information systems (CISs) is known to be an essential consideration in ensuring patient safety as well as integrating clinical flow. This study aimed to determine how usability and safety guidelines of CIS consider clinical workflow through a systematic review in terms of the target systems, methodology, and guideline components of relevant articles. Methods A literature search was conducted for articles published from 2000 to 2015 in PubMed, Cochrane, EMBASE, Web of Science, and CINAHL. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement method was employed. Articles containing recommendations, principles, and evaluation items for CIS usability and safety were included. The selected articles were classified according to article type, methodology, and target systems. Taking clinical workflow into consideration, the components of guidelines were extracted and classified. Results A total of 7,401 articles were identified by keyword search. From the 76 articles remaining after abstract screening, 15 were selected through full-text review. Literature review (n = 7) was the most common methodology, followed by expert opinions (n = 6). Computerized physician order entry (n = 6) was the most frequent system. Four articles considered the entire process of clinical tasks, and two articles considered the principles of the entire process of user interface affecting clinical workflow. Only two articles performed heuristic evaluations of CISs. Conclusions The usability and safety guidelines of CISs need improvement in guideline development methodology and with consideration of clinical workflow.
Collapse
Affiliation(s)
- Yura Lee
- Department of Biomedical Informatics, Asan Medical Center, Seoul, Korea
| | - Min-Young Jung
- Department of Biomedical Informatics, Asan Medical Center, Seoul, Korea
| | - Gee Won Shin
- Department of Industrial Engineering, Human Interface System Lab, Seoul National University, Seoul, Korea
| | - Sangwoo Bahn
- Department of Industrial and Management Systems Engineering, Kyung Hee University, Seoul, Korea
| | - Taezoon Park
- Department of Industrial and Information Systems Engineering, Soongsil University, Seoul, Korea
| | - Insook Cho
- Department of Nursing, Inha University, Incheon, Korea
| | - Jae-Ho Lee
- Department of Biomedical Informatics, Asan Medical Center, Seoul, Korea.,Department of Emergency Medicine, University of Ulsan Collage of Medicine, Seoul, Korea
| |
Collapse
|
45
|
Ghalibaf AK, Jangi M, Habibi MRM, Zangouei S, Khajouei R. Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method. Electron Physician 2018; 10:6682-6688. [PMID: 29881531 PMCID: PMC5984023 DOI: 10.19082/6682] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2017] [Accepted: 02/25/2018] [Indexed: 11/20/2022] Open
Abstract
Background and aim Obstetrics and gynecology information system is a critical component of the HIS in social security organization health centers. The objective of this study was to evaluate the usability of this system using the cognitive walkthrough method. Also, the present study provided a detailed formal description of how the cognitive usability evaluation can be applied and reported for a health care information system. Methods This study was conducted at the Mashhad University of Medical Sciences' usability lab from March 2016 to June 2017. A two-phase approach was used to conduct the cognitive walkthrough evaluation: preparatory and evaluation. The preparation was done in three stages: first, we investigated users' capabilities and background knowledge through a semi-structured interview. Second, the evaluation scenario was developed based on the most common tasks in routine workflow of users. Finally, each task was broken down into sequences of actions. In the evaluation phase, three usability experts independently assessed each action using a four-item checklist. Problems were categorized thematically and were reported from three different perspectives: Question-based, Task-based, and Evaluator-based. The data were then analyzed to understand the contribution of each task, along with its mean severity score. Results Evaluators' responses were compared and any conflict was resolved in an expert panel. A total of 116 usability problems were identified based on the consensus of the evaluators. Inadequate system feedback was found to be the main source of 43% of the problems, and resulted in users confusion. Conclusion Since the system was evaluated in its pilot implementation phase, there was an opportunity to prevent future potential usability problems. The use of a mixed quantitative and qualitative approach in this usability study provided a more comprehensive perspective of the system problems. This study provided a detailed description of conducting CW usability evaluation which can be used as a practical guide for future studies.
Collapse
Affiliation(s)
- Azadeh Kamel Ghalibaf
- Ph.D. Candidate, Department of Medical Informatics, Faculty of Medicine, Students Research Committee, Mashhad University of Medical Sciences, Mashhad, Iran
| | - Majid Jangi
- Ph.D. Candidate, Department of Medical Informatics, Faculty of Medicine, Students Research Committee, Mashhad University of Medical Sciences, Mashhad, Iran
| | - Mohammad Reza Mazaheri Habibi
- Ph.D. Candidate, Department of Medical Informatics, Faculty of Medicine, Students Research Committee, Mashhad University of Medical Sciences, Mashhad, Iran
| | - Sara Zangouei
- M.Sc. of Computer Engineering, Azad University of Mashhad, Mashhad, Iran
| | - Reza Khajouei
- Ph.D. in Medical Informatics, Medical Informatics Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran.,Ph.D. in Medical Informatics, Department of Health Information Sciences, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Iran
| |
Collapse
|
46
|
Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic Health Record Usability Issues and Potential Contribution to Patient Harm. JAMA 2018; 319:1276-1278. [PMID: 29584833 PMCID: PMC5885839 DOI: 10.1001/jama.2018.1171] [Citation(s) in RCA: 147] [Impact Index Per Article: 24.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
This study analyzed patient safety reports in and near Pennsylvania from 2013 through 2016 to identify those that contained explicit language associating possible patient harm with an electronic health record usability issue.
Collapse
Affiliation(s)
- Jessica L Howe
- National Center for Human Factors in Healthcare, MedStar Health, Washington, DC
| | - Katharine T Adams
- National Center for Human Factors in Healthcare, MedStar Health, Washington, DC
| | - A Zachary Hettinger
- National Center for Human Factors in Healthcare, MedStar Health, Washington, DC
| | - Raj M Ratwani
- National Center for Human Factors in Healthcare, MedStar Health, Washington, DC
| |
Collapse
|
47
|
Grout RW, Cheng ER, Carroll AE, Bauer NS, Downs SM. A six-year repeated evaluation of computerized clinical decision support system user acceptability. Int J Med Inform 2018; 112:74-81. [PMID: 29500025 DOI: 10.1016/j.ijmedinf.2018.01.011] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2017] [Revised: 01/05/2018] [Accepted: 01/15/2018] [Indexed: 12/22/2022]
Abstract
OBJECTIVE Long-term acceptability among computerized clinical decision support system (CDSS) users in pediatrics is unknown. We examine user acceptance patterns over six years of our continuous computerized CDSS integration and updates. MATERIALS AND METHODS Users of Child Health Improvement through Computer Automation (CHICA), a CDSS integrated into clinical workflows and used in several urban pediatric community clinics, completed annual surveys including 11 questions covering user acceptability. We compared responses across years within a single healthcare system and between two healthcare systems. We used logistic regression to assess the odds of a favorable response to each question by survey year, clinic role, part-time status, and frequency of CHICA use. RESULTS Data came from 380 completed surveys between 2011 and 2016. Responses were significantly more favorable for all but one measure by 2016 (OR range 2.90-12.17, all p < 0.01). Increasing system maturity was associated with improved perceived function of CHICA (OR range 4.24-7.58, p < 0.03). User familiarity was positively associated with perceived CDSS function (OR range 3.44-8.17, p < 0.05) and usability (OR range 9.71-15.89, p < 0.01) opinions. CONCLUSION We present a long-term, repeated follow-up of user acceptability of a CDSS. Favorable opinions of the CDSS were more likely in frequent users, physicians and advanced practitioners, and full-time workers. CHICA acceptability increased as it matured and users become more familiar with it. System quality improvement, user support, and patience are important in achieving wide-ranging, sustainable acceptance of CDSS.
Collapse
Affiliation(s)
- Randall W Grout
- Children's Health Services Research, Department of Pediatrics, School of Medicine, Indiana University, Indianapolis, IN, USA.
| | - Erika R Cheng
- Children's Health Services Research, Department of Pediatrics, School of Medicine, Indiana University, Indianapolis, IN, USA
| | - Aaron E Carroll
- Pediatric and Adolescent Comparative Effectiveness Research, Department of Pediatrics, School of Medicine, Indiana University, Indianapolis, IN, USA
| | - Nerissa S Bauer
- Children's Health Services Research, Department of Pediatrics, School of Medicine, Indiana University, Indianapolis, IN, USA
| | - Stephen M Downs
- Children's Health Services Research, Department of Pediatrics, School of Medicine, Indiana University, Indianapolis, IN, USA
| |
Collapse
|
48
|
Russ AL, Saleem JJ. Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inform 2018; 78:123-133. [DOI: 10.1016/j.jbi.2018.01.001] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2017] [Revised: 12/19/2017] [Accepted: 01/08/2018] [Indexed: 11/15/2022]
|
49
|
Lintern G, Motavalli A. Healthcare information systems: the cognitive challenge. BMC Med Inform Decis Mak 2018; 18:3. [PMID: 29325563 PMCID: PMC5765659 DOI: 10.1186/s12911-018-0584-z] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2017] [Accepted: 01/02/2018] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Healthcare work is, to a considerable extent, cognitive. Subsequently, the analysis and the design of supporting technology must be sensitive to the cognitive and adaptive demands of the work and to the cognitive strategies employed by healthcare practitioners. Despite the vital role that cognition plays in healthcare work, current technocentric design approaches for healthcare technology do not account for it, failing to observe it during analysis and failing to develop support for it during design. MAIN BODY By review and analysis of case studies, we show that healthcare systems developed without input from cognitive analysis and cognitive design fail to take account of important healthcare work processes and workflows. In contrast, systems developed with a cognitively-focused design strategy demonstrate how it is possible to introduce technology that supports and enhances the work strategies of those engaged in patient care. CONCLUSION Significant problems emerge when technological support systems are developed without any serious and comprehensive attempt to understand the cognitive capabilities and skills deployed by those involved in patient care. In contrast, significant benefits accrue from taking full account of those cognitive capabilities and skills. Subsequently, the design and development of supporting technology must be sensitive to the cognitive demands of the work and the cognitive strategies employed by healthcare practitioners.
Collapse
Affiliation(s)
- Gavan Lintern
- Monash University Accident Research Centre, Building 70, Monash, VIC, 3800, Australia.
| | - Al Motavalli
- Department of Anaesthesia, The Northern Hospital, 185 Cooper St, Epping, VIC, 3076, Australia
| |
Collapse
|
50
|
Wang Y, Wang L, Rastegar-Mojarad M, Moon S, Shen F, Afzal N, Liu S, Zeng Y, Mehrabi S, Sohn S, Liu H. Clinical information extraction applications: A literature review. J Biomed Inform 2018; 77:34-49. [PMID: 29162496 PMCID: PMC5771858 DOI: 10.1016/j.jbi.2017.11.011] [Citation(s) in RCA: 316] [Impact Index Per Article: 52.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2017] [Revised: 11/01/2017] [Accepted: 11/17/2017] [Indexed: 12/24/2022]
Abstract
BACKGROUND With the rapid adoption of electronic health records (EHRs), it is desirable to harvest information and knowledge from EHRs to support automated systems at the point of care and to enable secondary use of EHRs for clinical and translational research. One critical component used to facilitate the secondary use of EHR data is the information extraction (IE) task, which automatically extracts and encodes clinical information from text. OBJECTIVES In this literature review, we present a review of recent published research on clinical information extraction (IE) applications. METHODS A literature search was conducted for articles published from January 2009 to September 2016 based on Ovid MEDLINE In-Process & Other Non-Indexed Citations, Ovid MEDLINE, Ovid EMBASE, Scopus, Web of Science, and ACM Digital Library. RESULTS A total of 1917 publications were identified for title and abstract screening. Of these publications, 263 articles were selected and discussed in this review in terms of publication venues and data sources, clinical IE tools, methods, and applications in the areas of disease- and drug-related studies, and clinical workflow optimizations. CONCLUSIONS Clinical IE has been used for a wide range of applications, however, there is a considerable gap between clinical studies using EHR data and studies using clinical IE. This study enabled us to gain a more concrete understanding of the gap and to provide potential solutions to bridge this gap.
Collapse
Affiliation(s)
- Yanshan Wang
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Liwei Wang
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Majid Rastegar-Mojarad
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Sungrim Moon
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Feichen Shen
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Naveed Afzal
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Sijia Liu
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Yuqun Zeng
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Saeed Mehrabi
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Sunghwan Sohn
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Hongfang Liu
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.
| |
Collapse
|