1
|
Castner N, Arsiwala-Scheppach L, Mertens S, Krois J, Thaqi E, Kasneci E, Wahl S, Schwendicke F. Expert gaze as a usability indicator of medical AI decision support systems: a preliminary study. NPJ Digit Med 2024; 7:199. [PMID: 39068241 PMCID: PMC11283514 DOI: 10.1038/s41746-024-01192-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2023] [Accepted: 07/12/2024] [Indexed: 07/30/2024] Open
Abstract
Given the current state of medical artificial intelligence (AI) and perceptions towards it, collaborative systems are becoming the preferred choice for clinical workflows. This work aims to address expert interaction with medical AI support systems to gain insight towards how these systems can be better designed with the user in mind. As eye tracking metrics have been shown to be robust indicators of usability, we employ them for evaluating the usability and user interaction with medical AI support systems. We use expert gaze to assess experts' interaction with an AI software for caries detection in bitewing x-ray images. We compared standard viewing of bitewing images without AI support versus viewing where AI support could be freely toggled on and off. We found that experts turned the AI on for roughly 25% of the total inspection task, and generally turned it on halfway through the course of the inspection. Gaze behavior showed that when supported by AI, more attention was dedicated to user interface elements related to the AI support, with more frequent transitions from the image itself to these elements. When considering that expert visual strategy is already optimized for fast and effective image inspection, such interruptions in attention can lead to increased time needed for the overall assessment. Gaze analysis provided valuable insights into an AI's usability for medical image inspection. Further analyses of these tools and how to delineate metrical measures of usability should be developed.
Collapse
Affiliation(s)
- Nora Castner
- Carl Zeiss Vision International GmbH, Tübingen, Germany.
- University of Tübingen, Tübingen, Germany.
| | | | - Sarah Mertens
- Charité - Univesitätsmedizin, Oral Diagnostics, Digital Health and Services Research, Berlin, Germany
| | - Joachim Krois
- Charité - Univesitätsmedizin, Oral Diagnostics, Digital Health and Services Research, Berlin, Germany
| | - Enkeleda Thaqi
- Technical University of Munich, Human-Centered Technologies for Learning, Munich, Germany
| | - Enkelejda Kasneci
- Technical University of Munich, Human-Centered Technologies for Learning, Munich, Germany
| | - Siegfried Wahl
- Carl Zeiss Vision International GmbH, Tübingen, Germany
- Institute for Ophthalmic Research, University of Tübingen, Tübingen, Germany
| | - Falk Schwendicke
- Ludwig Maximilian University, Operative, Preventative and Pediatric Dentistry and Periodontology, Munich, Germany
| |
Collapse
|
2
|
Anders C, Moorthy P, Svensson L, Müller J, Heinze O, Knaup P, Wallwiener M, Deutsch TM, Le TV, Weinert L. Usability and User Experience of an mHealth App for Therapy Support of Patients With Breast Cancer: Mixed Methods Study Using Eye Tracking. JMIR Hum Factors 2024; 11:e50926. [PMID: 38441959 PMCID: PMC10951836 DOI: 10.2196/50926] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 11/30/2023] [Accepted: 12/17/2023] [Indexed: 03/07/2024] Open
Abstract
BACKGROUND Early identification of quality of life (QoL) loss and side effects is a key challenge in breast cancer therapy. Digital tools can be helpful components of therapeutic support. Enable, a smartphone app, was used in a multicenter, prospective randomized controlled trial in 3 breast cancer centers. The app simultaneously serves as a therapy companion (eg, by displaying appointments), a tool for documenting QoL (eg, by enabling data collection for QoL questionnaires), and documentation of patient-reported side effects. The need for digital tools is continually rising. However, evidence of the effects of long-term use of mobile health (mHealth) apps in aftercare for patients with breast cancer is limited. Therefore, evaluating the usability and understanding the user experience of this mHealth app could potentially contribute valuable insights in this field. OBJECTIVE A usability study was conducted to explore how patients with breast cancer receiving neoadjuvant, adjuvant, or palliative outpatient treatment rated their engagement with the app , the user experience, and the benefits of using the app. METHODS A mixed methods approach was chosen to combine subjective and objective measures, including an eye-tracking procedure, a standardized usability questionnaire (mHealth App Usability Questionnaire), and semistructured interviews. Participants were surveyed twice during the study period. Interviews were transcribed verbatim and analyzed using thematic analysis. Analysis of the eye-tracking data was carried out using the tracker-integrated software. Descriptive analysis was conducted for the quantitative data. RESULTS The mHealth App Usability Questionnaire results (n=105) indicated good overall usability for 2 different time points (4 wk: mean 89.15, SD 9.65; 20 wk: mean 85.57, SD 12.88). The qualitative analysis of the eye-tracking recordings (n=10) and interviews (n=16) showed that users found the Enable app easy to use. The design of the app, information about therapies and side effects, and usefulness of the app as a therapy companion were rated positively. Additionally, participants contributed requests for additional app features and suggestions for improving the content and usability of the app. Relevant themes included optimization of the appointment feature, updating the app's content regularly, and self-administration. In contrast to the app's current passive method of operation, participants expressed a desire for more active engagement through messaging, alarms, or emails. CONCLUSIONS The results of this study demonstrate the good usability of the Enable app as well as the potential for further development. We concluded from patients' feedback and requests that mHealth apps could benefit from giving patients a more active role (eg, being able to actively document side effects as they occur). Additionally, regular updates of app content could further contribute to encouraging continued use of mHealth apps. Our findings may also assist other researchers in tailoring their mHealth apps to the actual needs of patients undergoing breast cancer therapy.
Collapse
Affiliation(s)
- Carolin Anders
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Preetha Moorthy
- Department of Biomedical Informatics at the Center for Preventive Medicine and Digital Health (CPD), Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | - Laura Svensson
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Julia Müller
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Oliver Heinze
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Petra Knaup
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Markus Wallwiener
- Department of Obstetrics and Gynecology, Heidelberg University Hospital, Heidelberg, Germany
| | - Thomas M Deutsch
- Department of Obstetrics and Gynecology, Heidelberg University Hospital, Heidelberg, Germany
| | - Thao-Vy Le
- Department of Obstetrics and Gynecology, Heidelberg University Hospital, Heidelberg, Germany
| | - Lina Weinert
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
- Section for Oral Health, Heidelberg Institute of Global Health, Heidelberg University Hospital, Heidelberg, Germany
| |
Collapse
|
3
|
Keszthelyi D, Gaudet-Blavignac C, Bjelogrlic M, Lovis C. Patient Information Summarization in Clinical Settings: Scoping Review. JMIR Med Inform 2023; 11:e44639. [PMID: 38015588 DOI: 10.2196/44639] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 03/15/2023] [Accepted: 07/25/2023] [Indexed: 11/29/2023] Open
Abstract
BACKGROUND Information overflow, a common problem in the present clinical environment, can be mitigated by summarizing clinical data. Although there are several solutions for clinical summarization, there is a lack of a complete overview of the research relevant to this field. OBJECTIVE This study aims to identify state-of-the-art solutions for clinical summarization, to analyze their capabilities, and to identify their properties. METHODS A scoping review of articles published between 2005 and 2022 was conducted. With a clinical focus, PubMed and Web of Science were queried to find an initial set of reports, later extended by articles found through a chain of citations. The included reports were analyzed to answer the questions of where, what, and how medical information is summarized; whether summarization conserves temporality, uncertainty, and medical pertinence; and how the propositions are evaluated and deployed. To answer how information is summarized, methods were compared through a new framework "collect-synthesize-communicate" referring to information gathering from data, its synthesis, and communication to the end user. RESULTS Overall, 128 articles were included, representing various medical fields. Exclusively structured data were used as input in 46.1% (59/128) of papers, text in 41.4% (53/128) of articles, and both in 10.2% (13/128) of papers. Using the proposed framework, 42.2% (54/128) of the records contributed to information collection, 27.3% (35/128) contributed to information synthesis, and 46.1% (59/128) presented solutions for summary communication. Numerous summarization approaches have been presented, including extractive (n=13) and abstractive summarization (n=19); topic modeling (n=5); summary specification (n=11); concept and relation extraction (n=30); visual design considerations (n=59); and complete pipelines (n=7) using information extraction, synthesis, and communication. Graphical displays (n=53), short texts (n=41), static reports (n=7), and problem-oriented views (n=7) were the most common types in terms of summary communication. Although temporality and uncertainty information were usually not conserved in most studies (74/128, 57.8% and 113/128, 88.3%, respectively), some studies presented solutions to treat this information. Overall, 115 (89.8%) articles showed results of an evaluation, and methods included evaluations with human participants (median 15, IQR 24 participants): measurements in experiments with human participants (n=31), real situations (n=8), and usability studies (n=28). Methods without human involvement included intrinsic evaluation (n=24), performance on a proxy (n=10), or domain-specific tasks (n=11). Overall, 11 (8.6%) reports described a system deployed in clinical settings. CONCLUSIONS The scientific literature contains many propositions for summarizing patient information but reports very few comparisons of these proposals. This work proposes to compare these algorithms through how they conserve essential aspects of clinical information and through the "collect-synthesize-communicate" framework. We found that current propositions usually address these 3 steps only partially. Moreover, they conserve and use temporality, uncertainty, and pertinent medical aspects to varying extents, and solutions are often preliminary.
Collapse
Affiliation(s)
- Daniel Keszthelyi
- Division of Medical Information Sciences, University Hospitals of Geneva, Geneva, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland
| | - Christophe Gaudet-Blavignac
- Division of Medical Information Sciences, University Hospitals of Geneva, Geneva, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland
| | - Mina Bjelogrlic
- Division of Medical Information Sciences, University Hospitals of Geneva, Geneva, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland
| | - Christian Lovis
- Division of Medical Information Sciences, University Hospitals of Geneva, Geneva, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland
| |
Collapse
|
4
|
Tokgöz P, Hafner J, Dockweiler C. Factors influencing the implementation of decision support systems for antibiotic prescription in hospitals: a systematic review. BMC Med Inform Decis Mak 2023; 23:27. [PMID: 36747193 PMCID: PMC9903563 DOI: 10.1186/s12911-023-02124-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2022] [Accepted: 01/30/2023] [Indexed: 02/08/2023] Open
Abstract
BACKGROUND Antibiotic resistance is a major health threat. Inappropriate antibiotic use has been shown to be an important determinant of the emergence of antibiotic resistance. Decision support systems for antimicrobial management can support clinicians to optimize antibiotic prescription. OBJECTIVE The aim of this systematic review is to identify factors influencing the implementation of decision support systems for antibiotic prescription in hospitals. METHODS A systematic search of factors impeding or facilitating successful implementation of decision support systems for antibiotic prescription was performed in January 2022 in the databases PubMed, Web of Science and The Cochrane Library. Only studies were included which comprised decision support systems in hospitals for prescribing antibiotic therapy, published in English with a qualitative, quantitative or mixed-methods study design and between 2011 and 2021. Factors influencing the implementation were identified through text analysis by two reviewers. RESULTS A total of 14 publications were identified matching the inclusion criteria. The majority of factors relate to technological and organizational aspects of decision support system implementation. Some factors include the integration of the decision support systems into existing systems, system design, consideration of potential end-users as well as training and support for end-users. In addition, user-related factors, like user attitude towards the system, computer literacy and prior experience with the system seem to be important for successful implementation of decision support systems for antibiotic prescription in hospitals. CONCLUSION The results indicate a broad spectrum of factors of decision support system implementation for antibiotic prescription and contributes to the literature by identifying important organizational as well as user-related factors. Wider organizational dimensions as well as the interaction between user and technology appear important for supporting implementation.
Collapse
Affiliation(s)
- Pinar Tokgöz
- School of Life Sciences, Department Digital Health Sciences and Biomedicine, Professorship of Digital Public Health, University of Siegen, 57068, Siegen, Germany.
| | - Jessica Hafner
- grid.5836.80000 0001 2242 8751School of Life Sciences, Department Digital Health Sciences and Biomedicine, Professorship of Digital Public Health, University of Siegen, 57068 Siegen, Germany
| | - Christoph Dockweiler
- grid.5836.80000 0001 2242 8751School of Life Sciences, Department Digital Health Sciences and Biomedicine, Professorship of Digital Public Health, University of Siegen, 57068 Siegen, Germany
| |
Collapse
|
5
|
Wronikowska MW, Malycha J, Morgan LJ, Westgate V, Petrinic T, Young JD, Watkinson PJ. Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems. J Eval Clin Pract 2021; 27:1403-1416. [PMID: 33982356 PMCID: PMC9438452 DOI: 10.1111/jep.13582] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/18/2020] [Revised: 04/19/2021] [Accepted: 04/20/2021] [Indexed: 01/01/2023]
Abstract
BACKGROUND AND OBJECTIVES Electronic healthcare records have become central to patient care. Evaluation of new systems include a variety of usability evaluation methods or usability metrics (often referred to interchangeably as usability components or usability attributes). This study reviews the breadth of usability evaluation methods, metrics, and associated measurement techniques that have been reported to assess systems designed for hospital staff to assess inpatient clinical condition. METHODS Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology, we searched Medline, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, and Open Grey from 1986 to 2019. For included studies, we recorded usability evaluation methods or usability metrics as appropriate, and any measurement techniques applied to illustrate these. We classified and described all usability evaluation methods, usability metrics, and measurement techniques. Study quality was evaluated using a modified Downs and Black checklist. RESULTS The search identified 1336 studies. After abstract screening, 130 full texts were reviewed. In the 51 included studies 11 distinct usability evaluation methods were identified. Within these usability evaluation methods, seven usability metrics were reported. The most common metrics were ISO9241-11 and Nielsen's components. An additional "usefulness" metric was reported in almost 40% of included studies. We identified 70 measurement techniques used to evaluate systems. Overall study quality was reflected in a mean modified Downs and Black checklist score of 6.8/10 (range 1-9) 33% studies classified as "high-quality" (scoring eight or higher), 51% studies "moderate-quality" (scoring 6-7), and the remaining 16% (scoring below five) were "low-quality." CONCLUSION There is little consistency within the field of electronic health record systems evaluation. This review highlights the variability within usability methods, metrics, and reporting. Standardized processes may improve evaluation and comparison electronic health record systems and improve their development and implementation.
Collapse
Affiliation(s)
| | - James Malycha
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
- Department of Acute Care MedicineUniversity of AdelaideAdelaideAustralia
| | - Lauren J. Morgan
- Nuffield Department of Surgical SciencesUniversity of Oxford, John Radcliffe HospitalOxfordUK
| | - Verity Westgate
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
| | - Tatjana Petrinic
- Bodleian Health Care LibrariesJohn Radcliffe Hospital, University of OxfordOxfordUK
| | - J Duncan Young
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
| | - Peter J. Watkinson
- Critical Care Research Group, Nuffield Department of Clinical NeurosciencesUniversity of OxfordOxfordUK
| |
Collapse
|
6
|
Kołodziej P, Tuszyńska-Bogucka W, Dzieńkowski M, Bogucki J, Kocki J, Milosz M, Kocki M, Reszka P, Kocki W, Bogucka-Kocka A. Eye Tracking-An Innovative Tool in Medical Parasitology. J Clin Med 2021; 10:jcm10132989. [PMID: 34279473 PMCID: PMC8268455 DOI: 10.3390/jcm10132989] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2021] [Revised: 06/29/2021] [Accepted: 06/29/2021] [Indexed: 11/16/2022] Open
Abstract
The innovative Eye Movement Modelling Examples (EMMEs) method can be used in medicine as an educational training tool for the assessment and verification of students and professionals. Our work was intended to analyse the possibility of using eye tracking tools to verify the skills and training of people engaged in laboratory medicine on the example of parasitological diagnostics. Professionally active laboratory diagnosticians working in a multi-profile laboratory (non-parasitological) (n = 16), laboratory diagnosticians no longer working in this profession (n = 10), and medical analyst students (n = 56), participated in the study. The studied group analysed microscopic images of parasitological preparations made with the cellSens Dimension Software (Olympus) system. Eye activity parameters were obtained using a stationary, video-based eye tracker Tobii TX300 which has a 3-ms temporal resolution. Eye movement activity parameters were analysed along with time parameters. The results of our studies have shown that the eye tracking method is a valuable tool for the analysis of parasitological preparations. Detailed quantitative and qualitative analysis confirmed that the EMMEs method may facilitate learning of the correct microscopic image scanning path. The analysis of the results of our studies allows us to conclude that the EMMEs method may be a valuable tool in the preparation of teaching materials in virtual microscopy. These teaching materials generated with the use of eye tracking, prepared by experienced professionals in the field of laboratory medicine, can be used during various training, simulations and courses in medical parasitology and contribute to the verification of education results, professional skills, and elimination of errors in parasitological diagnostics.
Collapse
Affiliation(s)
- Przemysław Kołodziej
- Chair and Department of Biology and Genetics, Medical University of Lublin, 20-093 Lublin, Poland;
- Correspondence: ; Tel.: +48-814-487-234
| | | | - Mariusz Dzieńkowski
- Department of Computer Science, Lublin University of Technology, 20-618 Lublin, Poland; (M.D.); (M.M.)
| | - Jacek Bogucki
- Department of Organic Chemistry, Medical University of Lublin, 20-093 Lublin, Poland;
| | - Janusz Kocki
- Department of Clinical Genetics, Medical University of Lublin, 20-080 Lublin, Poland;
| | - Marek Milosz
- Department of Computer Science, Lublin University of Technology, 20-618 Lublin, Poland; (M.D.); (M.M.)
| | - Marcin Kocki
- Scientific Circle at Department of Clinical Genetics, Medical University of Lublin, 20-080 Lublin, Poland; (M.K.); (P.R.)
| | - Patrycja Reszka
- Scientific Circle at Department of Clinical Genetics, Medical University of Lublin, 20-080 Lublin, Poland; (M.K.); (P.R.)
| | - Wojciech Kocki
- Department of Architecture and Urban Planning, Lublin University of Technology, 20-618 Lublin, Poland;
| | - Anna Bogucka-Kocka
- Chair and Department of Biology and Genetics, Medical University of Lublin, 20-093 Lublin, Poland;
| |
Collapse
|
7
|
Pestana M, Pereira R, Moro S. Improving Health Care Management in Hospitals Through a Productivity Dashboard. J Med Syst 2020; 44:87. [PMID: 32166499 DOI: 10.1007/s10916-020-01546-1] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2019] [Accepted: 02/18/2020] [Indexed: 10/24/2022]
Abstract
Health information systems have been developed to help hospital managers steer daily operations, including key performance indicators (KPIs) for monitoring on a time-aggregated basis. Yet, current literature lacks in proposals of productivity dashboards to assist hospitals stakeholders. This research focuses on two related problems: (1) hospital organizations need access to productivity information to improve access to services; and (2) managers need productivity information to optimize resource allocation. This research consists in the development of dashboards to monitor information obtained from a hospital organization to support decision makers. To develop and evaluate the productivity dashboard, the Design Science Research (DSR) methodology was adopted. The dashboard was evaluated by stakeholders of a large Portuguese hospital who contributed to iteratively improving its design toward a useful decision support tool. Additionally, it was ascertained that monitoring productivity needs more study and that the dashboards on these themes are valuable assets at a monitoring level and subsequent decision-making process.
Collapse
Affiliation(s)
- Miguel Pestana
- DCTI, ISCTE-Instituto Universitário de Lisboa, Line 1: Av. das Forças Armadas, 1649-026, Lisbon, Portugal
| | - Ruben Pereira
- DCTI, ISCTE-Instituto Universitário de Lisboa, Line 1: Av. das Forças Armadas, 1649-026, Lisbon, Portugal.
| | - Sérgio Moro
- DCTI, ISCTE-Instituto Universitário de Lisboa, Line 1: Av. das Forças Armadas, 1649-026, Lisbon, Portugal
| |
Collapse
|
8
|
Pollack AH, Pratt W. Association of Health Record Visualizations With Physicians' Cognitive Load When Prioritizing Hospitalized Patients. JAMA Netw Open 2020; 3:e1919301. [PMID: 31940040 PMCID: PMC6991320 DOI: 10.1001/jamanetworkopen.2019.19301] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/29/2022] Open
Abstract
IMPORTANCE Current electronic health records (EHRs) contribute to increased physician cognitive workload when completing clinical tasks. OBJECTIVE To assess the association of different design features of an EHR-based information visualization tool with the cognitive load of physicians during the clinical prioritization process. DESIGN, SETTING, AND PARTICIPANTS This cross-sectional study included a convenience sample of 29 attending physicians at Seattle Children's Hospital, a large tertiary academic pediatric hospital. Data collection took place from August 2017 through October 2017, and analysis occurred from August to October 2018. EXPOSURE Physician participants used 3 prototypes with novel visualizations of simulated EHR data that highlighted 1 of 3 key patient characteristics, as follows: (1) acuity, (2) clinical problem list, and (3) clinical change. MAIN OUTCOMES AND MEASURES Cognitive workload was measured using the NASA Task Load Index (TLX) scale (range, 1-100, with lower scores indicating lower cognitive workload). Cognitive workload was assessed for the 2 following clinical prioritization tasks: (1) finding information for a specific patient and (2) comparing results among patients for each prototype. Participants ranked 5 hypothetical patients from having the highest to the lowest priority in each design. RESULTS A total of 29 physician participants (15 [52%] men; 14 [48%] women; mean [range] age, 43 [35-58] years; mean [range] time in practice, 11 [3-30] years) completed the study. For task 1, the prototype highlighting clinical change was associated with lower median (interquartile range) NASA TLX scores compared with the prototype highlighting acuity (30.3 [15.2-41.6] vs 48.5 [18.7-59.3]; P = .02). For task 2, the prototype highlighting clinical change was associated with lower median (interquartile range) NASA TLX scores compared with the prototype highlighting the clinical problem list (29.1 [16.3-50.8] vs 43.5 [26.6-55.9]; P = .02). The prototype highlighting clinical change had the lowest TLX score in 17 of 29 rankings (59%) for task 1 (χ24 = 24.4; P < .001) and 18 of 29 rankings (62%) for task 2 (χ24 = 17.2; P = .002). CONCLUSIONS AND RELEVANCE In this study, well-designed EHR-based information visualizations that highlighted and featured clinically meaningful information patterns significantly reduced physician cognitive workload when prioritizing patient needs.
Collapse
Affiliation(s)
- Ari H Pollack
- Division of Nephrology, Seattle Children's Hospital, Seattle, Washington
- Department of Pediatrics, University of Washington School of Medicine, Seattle
| | - Wanda Pratt
- Information School, University of Washington, Seattle
| |
Collapse
|
9
|
Wright MC, Borbolla D, Waller RG, Del Fiol G, Reese T, Nesbitt P, Segall N. Critical care information display approaches and design frameworks: A systematic review and meta-analysis. J Biomed Inform 2019; 3:100041. [PMID: 31423485 PMCID: PMC6696941 DOI: 10.1016/j.yjbinx.2019.100041] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2018] [Revised: 06/10/2019] [Accepted: 06/16/2019] [Indexed: 11/06/2022]
Abstract
OBJECTIVE To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians. METHODS We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author. RESULTS Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible. CONCLUSION Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.
Collapse
Affiliation(s)
- Melanie C. Wright
- Trinity Health, Livonia, MI, USA
- Saint Alphonsus Regional Medical Center, Boise, ID, USA
| | - Damian Borbolla
- Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | | | | | - Thomas Reese
- Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - Paige Nesbitt
- Saint Alphonsus Regional Medical Center, Boise, ID, USA
| | - Noa Segall
- Anesthesiology, Duke University, Durham, NC, USA
| |
Collapse
|
10
|
Rivas C, Tkacz D, Antao L, Mentzakis E, Gordon M, Anstee S, Giordano R. Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study. HEALTH SERVICES AND DELIVERY RESEARCH 2019. [DOI: 10.3310/hsdr07230] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
BackgroundPatient experience surveys (PESs) often include informative free-text comments, but with no way of systematically, efficiently and usefully analysing and reporting these. The National Cancer Patient Experience Survey (CPES), used to model the approach reported here, generates > 70,000 free-text comments annually.Main aimTo improve the use and usefulness of PES free-text comments in driving health service changes that improve the patient experience.Secondary aims(1) To structure CPES free-text comments using rule-based information retrieval (IR) (‘text engineering’), drawing on health-care domain-specific gazetteers of terms, with in-built transferability to other surveys and conditions; (2) to display the results usefully for health-care professionals, in a digital toolkit dashboard display that drills down to the original free text; (3) to explore the usefulness of interdisciplinary mixed stakeholder co-design and consensus-forming approaches in technology development, ensuring that outputs have meaning for all; and (4) to explore the usefulness of Normalisation Process Theory (NPT) in structuring outputs for implementation and sustainability.DesignA scoping review, rapid review and surveys with stakeholders in health care (patients, carers, health-care providers, commissioners, policy-makers and charities) explored clinical dashboard design/patient experience themes. The findings informed the rules for the draft rule-based IR [developed using half of the 2013 Wales CPES (WCPES) data set] and prototype toolkit dashboards summarising PES data. These were refined following mixed stakeholder, concept-mapping workshops and interviews, which were structured to enable consensus-forming ‘co-design’ work. IR validation used the second half of the WCPES, with comparison against its manual analysis; transferability was tested using further health-care data sets. A discrete choice experiment (DCE) explored which toolkit features were preferred by health-care professionals, with a simple cost–benefit analysis. Structured walk-throughs with NHS managers in Wessex, London and Leeds explored usability and general implementation into practice.Key outcomesA taxonomy of ranked PES themes, a checklist of key features recommended for digital clinical toolkits, rule-based IR validation and transferability scores, usability, and goal-oriented, cost–benefit and marketability results. The secondary outputs were a survey, scoping and rapid review findings, and concordance and discordance between stakeholders and methods.Results(1) The surveys, rapid review and workshops showed that stakeholders differed in their understandings of the patient experience and priorities for change, but that they reached consensus on a shortlist of 19 themes; six were considered to be core; (2) the scoping review and one survey explored the clinical toolkit design, emphasising that such toolkits should be quick and easy to use, and embedded in workflows; the workshop discussions, the DCE and the walk-throughs confirmed this and foregrounded other features to form the toolkit design checklist; and (3) the rule-based IR, developed using noun and verb phrases and lookup gazetteers, was 86% accurate on the WCPES, but needs modification to improve this and to be accurate with other data sets. The DCE and the walk-through suggest that the toolkit would be well accepted, with a favourable cost–benefit ratio, if implemented into practice with appropriate infrastructure support.LimitationsSmall participant numbers and sampling bias across component studies. The scoping review studies mostly used top-down approaches and focused on professional dashboards. The rapid review of themes had limited scope, with no second reviewer. The IR needs further refinement, especially for transferability. New governance restrictions further limit immediate use.ConclusionsUsing a multidisciplinary, mixed stakeholder, use of co-design, proof of concept was shown for an automated display of patient experience free-text comments in a way that could drive health-care improvements in real time. The approach is easily modified for transferable application.Future workFurther exploration is needed of implementation into practice, transferable uses and technology development co-design approaches.FundingThe National Institute for Health Research Health Services and Delivery Research programme.
Collapse
Affiliation(s)
- Carol Rivas
- Faculty of Health Sciences, University of Southampton, Southampton, UK
- Department of Social Science Research Unit, University College London, London, UK
| | - Daria Tkacz
- Faculty of Health Sciences, University of Southampton, Southampton, UK
| | - Laurence Antao
- Faculty of Health Sciences, University of Southampton, Southampton, UK
| | - Emmanouil Mentzakis
- Economics within Social Sciences, University of Southampton, Southampton, UK
| | | | - Sydney Anstee
- Faculty of Health Sciences, University of Southampton, Southampton, UK
| | - Richard Giordano
- Faculty of Health Sciences, University of Southampton, Southampton, UK
| |
Collapse
|
11
|
Lin YL, Trbovich P, Kolodzey L, Nickel C, Guerguerian AM. Association of Data Integration Technologies With Intensive Care Clinician Performance: A Systematic Review and Meta-analysis. JAMA Netw Open 2019; 2:e194392. [PMID: 31125104 PMCID: PMC6632132 DOI: 10.1001/jamanetworkopen.2019.4392] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/02/2023] Open
Abstract
IMPORTANCE Sources of data in the intensive care setting are increasing exponentially, but the benefits of displaying multiparametric, high-frequency data are unknown. Decision making may not benefit from this technology if clinicians remain cognitively overburdened by poorly designed data integration and visualization technologies (DIVTs). OBJECTIVE To systematically review and summarize the published evidence on the association of user-centered DIVTs with intensive care clinician performance. DATA SOURCES MEDLINE, Embase, Cochrane Central Register of Controlled Trials, PsycINFO, and Web of Science were searched in May 2014 and January 2018. STUDY SELECTION Studies had 3 requirements: (1) the study tested a viable DIVT, (2) participants involved were intensive care clinicians, and (3) the study reported quantitative results associated with decision making in an intensive care setting. DATA EXTRACTION AND SYNTHESIS Of 252 records screened, 20 studies, published from 2004 to 2016, were included. The human factors framework to assess health technologies was applied to measure study completeness, and the Quality Assessment Instrument was used to assess the quality of the studies. PRISMA guidelines were adapted to conduct the systematic review and meta-analysis. MAIN OUTCOMES AND MEASURES Study completeness and quality; clinician performance; physical, mental, and temporal demand; effort; frustration; time to decision; and decision accuracy. RESULTS Of the 20 included studies, 16 were experimental studies with 410 intensive care clinician participants and 4 were survey-based studies with 1511 respondents. Scores for study completeness ranged from 27 to 43, with a maximum score of 47, and scores for study quality ranged from 46 to 79, with a maximum score of 90. Of 20 studies, DIVTs were evaluated in clinical settings in 2 studies (10%); time to decision was measured in 14 studies (70%); and decision accuracy was measured in 11 studies (55%). Measures of cognitive workload pooled in the meta-analysis suggested that any DIVT was an improvement over paper-based data in terms of self-reported performance, mental and temporal demand, and effort. With a maximum score of 22, median (IQR) mental demand scores for electronic display were 10 (7-13), tabular display scores were 8 (6.0-11.5), and novel visualization scores were 8 (6-12), compared with 17 (14-19) for paper. The median (IQR) temporal demand scores were also lower for all electronic visualizations compared with paper, with scores of 8 (6-11) for electronic display, 7 (6-11) for tabular and bar displays, 7 (5-11) for novel visualizations, and 16 (14.3-19.0) for paper. The median (IQR) performance scores improved for all electronic visualizations compared with paper (lower score indicates better self-reported performance), with scores of 6 (3-11) for electronic displays, 6 (4-11) for tabular and bar displays, 6 (4-11) for novel visualizations, and 14 (11-16) for paper. Frustration and physical demand domains of cognitive workload did not change, and differences between electronic displays were not significant. CONCLUSIONS AND RELEVANCE This review suggests that DIVTs are associated with increased integration and consistency of data. Much work remains to identify which visualizations effectively reduce cognitive workload to enhance decision making based on intensive care data. Standardizing human factors testing by developing a repository of open access benchmarked test protocols, using a set of outcome measures, scenarios, and data sets, may accelerate the design and selection of the most appropriate DIVT.
Collapse
Affiliation(s)
- Ying Ling Lin
- Institute of Biomaterials and Biomedical Engineering, Faculty of Engineering, University of Toronto, Toronto, Ontario, Canada
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
| | - Patricia Trbovich
- Institute of Biomaterials and Biomedical Engineering, Faculty of Engineering, University of Toronto, Toronto, Ontario, Canada
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Badeau Family Research Chair in Patient Safety and Quality Improvement, North York General Hospital, University of Toronto, Toronto, Ontario, Canada
| | - Lauren Kolodzey
- Institute of Biomaterials and Biomedical Engineering, Faculty of Engineering, University of Toronto, Toronto, Ontario, Canada
| | - Cheri Nickel
- Hospital Library and Archives, The Hospital for Sick Children, Toronto, Ontario, Canada
| | - Anne-Marie Guerguerian
- Institute of Biomaterials and Biomedical Engineering, Faculty of Engineering, University of Toronto, Toronto, Ontario, Canada
- Department of Critical Care Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
- Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, Ontario, Canada
- Neurosciences and Mental Health Program, Research Institute, The Hospital for Sick Children, Toronto, Ontario, Canada
| |
Collapse
|
12
|
Cornish L, Hill A, Horswill MS, Becker SI, Watson MO. Eye-tracking reveals how observation chart design features affect the detection of patient deterioration: An experimental study. APPLIED ERGONOMICS 2019; 75:230-242. [PMID: 30509531 DOI: 10.1016/j.apergo.2018.10.005] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/25/2017] [Revised: 10/09/2018] [Accepted: 10/14/2018] [Indexed: 06/09/2023]
Abstract
Particular design features intended to improve usability - including graphically displayed observations and integrated colour-based scoring-systems - have been shown to increase the speed and accuracy with which users of hospital observation charts detect abnormal patient observations. We used eye-tracking to evaluate two potential cognitive mechanisms underlying these effects. Novice chart-users completed a series of experimental trials in which they viewed patient data presented on one of three observation chart designs (varied within-subjects), and indicated which observation was abnormal (or that none were). A chart that incorporated both graphically displayed observations and an integrated colour-based scoring-system yielded faster, more accurate responses and fewer, shorter fixations than a graphical chart without a colour-based scoring-system. The latter, in turn, yielded the same advantages over a tabular chart (which incorporated neither design feature). These results suggest that both colour-based scoring-systems and graphically displayed observations improve search efficiency and reduce the cognitive resources required to process vital sign data.
Collapse
Affiliation(s)
- Lillian Cornish
- School of Psychology, The University of Queensland, St Lucia, Brisbane, Queensland, 4072, Australia
| | - Andrew Hill
- School of Psychology, The University of Queensland, St Lucia, Brisbane, Queensland, 4072, Australia; Clinical Skills Development Service, Metro North Hospital and Health Service, Herston, Brisbane, Queensland, 4006, Australia.
| | - Mark S Horswill
- School of Psychology, The University of Queensland, St Lucia, Brisbane, Queensland, 4072, Australia
| | - Stefanie I Becker
- School of Psychology, The University of Queensland, St Lucia, Brisbane, Queensland, 4072, Australia
| | - Marcus O Watson
- School of Psychology, The University of Queensland, St Lucia, Brisbane, Queensland, 4072, Australia; Clinical Skills Development Service, Metro North Hospital and Health Service, Herston, Brisbane, Queensland, 4006, Australia; School of Medicine, The University of Queensland, Herston, Brisbane, Queensland, 4006, Australia
| |
Collapse
|
13
|
Geographical mapping of advanced practice registered nurses in Louisiana: Leveraging the power of data display. J Am Assoc Nurse Pract 2018; 31:8-16. [PMID: 30383550 DOI: 10.1097/jxx.0000000000000091] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND AND PURPOSE Identifying the locations of advanced practice registered nurses (APRNs) in the state is challenging without data readily available, which can adversely affect stakeholder initiatives. The purpose of this project was to develop an online interactive map reflecting distribution of APRNs licensed by the Louisiana State Board of Nursing (LSBN), heightening APRN visibility and improving usability of data for policy, education, practice, and research initiatives. METHODS A descriptive design with development of a graphical interactive map showing all Louisiana parishes with integrated distribution summaries of APRNs was made. Work proceeded from paper to higher fidelity simulations to the final interactive web page. Compatibility with LSBN's web page and annual data upload were ensured. CONCLUSION The project was done in collaboration with LSBN, an academic institution, and a privately hired web designer. The principals of Louisiana State Board of Nursing approved the final product. The product meets and exceeds the expectations for mapping. Evaluation of use and usability will be done at the discretion of LSBN. IMPLICATIONS FOR PRACTICE The outcome of this project will serve to heighten visibility of the distribution of APRNs in the state of Louisiana, enhancing opportunities for APRNs and for patients. Furthermore, those interested in leveraging the information in policy, education, or research initiatives will be served.
Collapse
|
14
|
Backonja U, Haynes SC, Kim KK. Data Visualizations to Support Health Practitioners' Provision of Personalized Care for Patients With Cancer and Multiple Chronic Conditions: User-Centered Design Study. JMIR Hum Factors 2018; 5:e11826. [PMID: 30327290 PMCID: PMC6231796 DOI: 10.2196/11826] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2018] [Revised: 08/29/2018] [Accepted: 09/08/2018] [Indexed: 12/13/2022] Open
Abstract
BACKGROUND There exists a challenge of understanding and integrating various types of data collected to support the health of individuals with multiple chronic conditions engaging in cancer care. Data visualization has the potential to address this challenge and support personalized cancer care. OBJECTIVE The aim of the study was to assess the health care practitioners' perceptions of and feedback regarding visualizations developed to support the care of individuals with multiple chronic conditions engaging in cancer care. METHODS Medical doctors (n=4) and registered nurses (n=4) providing cancer care at an academic medical center in the western United States provided feedback on visualization mock-ups. Mock-up designs were guided by current health informatics and visualization literature and the Munzner Nested Model for Visualization Design. User-centered design methods, a mock patient persona, and a scenario were used to elicit insights from participants. Directed content analysis was used to identify themes from session transcripts. Means and SDs were calculated for health care practitioners' rankings of overview visualizations. RESULTS Themes identified were data elements, supportive elements, confusing elements, interpretation, and use of visualization. Overall, participants found the visualizations useful and with the potential to provide personalized care. Use of color, reference lines, and familiar visual presentations (calendars, line graphs) were noted as helpful in interpreting data. CONCLUSIONS Visualizations guided by a framework and literature can support health care practitioners' understanding of data for individuals with multiple chronic conditions engaged in cancer care. This understanding has the potential to support the provision of personalized care.
Collapse
Affiliation(s)
- Uba Backonja
- Nursing and Healthcare Leadership, University of Washington Tacoma, Tacoma, WA, United States.,Biomedical Informatics & Medical Education, University of Washington School of Medicine, Seattle, WA, United States
| | - Sarah C Haynes
- Betty Irene Moore School of Nursing, University of California Davis, Sacramento, CA, United States
| | - Katherine K Kim
- Betty Irene Moore School of Nursing, University of California Davis, Sacramento, CA, United States
| |
Collapse
|
15
|
Squellati R. Evidence-Based Practice in the Treatment for Antibiotic-Associated Diarrhea in the Intensive Care Unit. Crit Care Nurs Clin North Am 2017; 30:87-99. [PMID: 29413218 DOI: 10.1016/j.cnc.2017.10.008] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
Abstract
Unit nurses provide care to patients with serious health conditions. Often antibiotics are recommended to fight infections. Sometimes patients are on proton pump inhibitors (PPI). Antibiotics and PPIs may lead to diarrhea, causing the patient more discomfort, and possibly leading to a more serious infection. One serious infection is caused by Clostridium difficile, which causes death in some cases. About 75% of patients on antibiotics may not need antibiotics. Several studies showed less diarrhea in patients on probiotics. However, probiotics cause harm to some patients, which needs to be considered. Nurses need to ensure that antibiotic stewardship is followed.
Collapse
|
16
|
Johnson A, Khotskaya YB, Brusco L, Zeng J, Holla V, Bailey AM, Litzenburger BC, Sanchez N, Shufean MA, Piha-Paul S, Subbiah V, Hong D, Routbort M, Broaddus R, Mills Shaw KR, Mills GB, Mendelsohn J, Meric-Bernstam F. Clinical Use of Precision Oncology Decision Support. JCO Precis Oncol 2017; 2017. [PMID: 30320296 DOI: 10.1200/po.17.00036] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022] Open
Abstract
PURPOSE Precision oncology is hindered by the lack of decision support for determining the functional and therapeutic significance of genomic alterations in tumors and relevant clinically available options. To bridge this knowledge gap, we established a Precision Oncology Decision Support (PODS) team that provides annotations at the alteration-level and subsequently determined if clinical decision-making was influenced. METHODS Genomic alterations were annotated to determine actionability based on a variant's known or potential functional and/or therapeutic significance. The medical records of a subset of patients annotated in 2015 were manually reviewed to assess trial enrollment. A web-based survey was implemented to capture the reasons why genotype-matched therapies were not pursued. RESULTS PODS processed 1,669 requests for annotation of 4,084 alterations (2,254 unique) across 49 tumor types for 1,197 patients. 2,444 annotations for 669 patients included an actionable variant call: 32.5% actionable, 9.4% potentially, 29.7% unknown, 28.4% non-actionable. 66% of patients had at least one actionable/potentially actionable alteration. 20.6% (110/535) patients annotated enrolled on a genotype-matched trial. Trial enrolment was significantly higher for patients with actionable/potentially actionable alterations (92/333, 27.6%) than those with unknown (16/136, 11.8%) and non-actionable (2/66, 3%) alterations (p=0.00004). Actionable alterations in PTEN, PIK3CA, and ERBB2 most frequently led to enrollment on genotype-matched trials. Clinicians cited a variety of reasons why patients with actionable alterations did not enroll on trials. CONCLUSION Over half of alterations annotated were of unknown significance or non-actionable. Physicians were more likely to enroll a patient on a genotype-matched trial when an annotation supported actionability. Future studies are needed to demonstrate the impact of decision support on trial enrollment and oncologic outcomes.
Collapse
Affiliation(s)
- Amber Johnson
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Yekaterina B Khotskaya
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Lauren Brusco
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Jia Zeng
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Vijaykumar Holla
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Ann M Bailey
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Beate C Litzenburger
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Nora Sanchez
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Md Abu Shufean
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Sarina Piha-Paul
- Department of Investigational Cancer Therapeutics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Vivek Subbiah
- Department of Investigational Cancer Therapeutics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - David Hong
- Department of Investigational Cancer Therapeutics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Mark Routbort
- Department of Hematopathology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Russell Broaddus
- Department of Pathology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Kenna R Mills Shaw
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Gordon B Mills
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.,Department of Systems Biology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - John Mendelsohn
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.,Department of Genomic Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Funda Meric-Bernstam
- Sheikh Khalifa Bin Zayed Al Nahyan Institute for Personalized Cancer Therapy, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.,Department of Investigational Cancer Therapeutics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.,Department of Surgical Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| |
Collapse
|
17
|
Lin YL, Guerguerian AM, Tomasi J, Laussen P, Trbovich P. "Usability of data integration and visualization software for multidisciplinary pediatric intensive care: a human factors approach to assessing technology". BMC Med Inform Decis Mak 2017; 17:122. [PMID: 28806954 PMCID: PMC5557066 DOI: 10.1186/s12911-017-0520-7] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2017] [Accepted: 08/04/2017] [Indexed: 11/17/2022] Open
Abstract
Background Intensive care clinicians use several sources of data in order to inform decision-making. We set out to evaluate a new interactive data integration platform called T3™ made available for pediatric intensive care. Three primary functions are supported: tracking of physiologic signals, displaying trajectory, and triggering decisions, by highlighting data or estimating risk of patient instability. We designed a human factors study to identify interface usability issues, to measure ease of use, and to describe interface features that may enable or hinder clinical tasks. Methods Twenty-two participants, consisting of bedside intensive care physicians, nurses, and respiratory therapists, tested the T3™ interface in a simulation laboratory setting. Twenty tasks were performed with a true-to-setting, fully functional, prototype, populated with physiological and therapeutic intervention patient data. Primary data visualization was time series and secondary visualizations were: 1) shading out-of-target values, 2) mini-trends with exaggerated maxima and minima (sparklines), and 3) bar graph of a 16-parameter indicator. Task completion was video recorded and assessed using a use error rating scale. Usability issues were classified in the context of task and type of clinician. A severity rating scale was used to rate potential clinical impact of usability issues. Results Time series supported tracking a single parameter but partially supported determining patient trajectory using multiple parameters. Visual pattern overload was observed with multiple parameter data streams. Automated data processing using shading and sparklines was often ignored but the 16-parameter data reduction algorithm, displayed as a persistent bar graph, was visually intuitive. However, by selecting or automatically processing data, triggering aids distorted the raw data that clinicians use regularly. Consequently, clinicians could not rely on new data representations because they did not know how they were established or derived. Conclusions Usability issues, observed through contextual use, provided directions for tangible design improvements of data integration software that may lessen use errors and promote safe use. Data-driven decision making can benefit from iterative interface redesign involving clinician-users in simulated environments. This study is a first step in understanding how software can support clinicians’ decision making with integrated continuous monitoring data. Importantly, testing of similar platforms by all the different disciplines who may become clinician users is a fundamental step necessary to understand the impact on clinical outcomes of decision aids. Electronic supplementary material The online version of this article (doi:10.1186/s12911-017-0520-7) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Ying Ling Lin
- Institute of Biomaterials and Biomedical Engineering, University of Toronto, Rosebrugh Building (RS), 164 College Street, Room 407, Toronto, ON, M5S 3G9, Canada.,Department of Critical Care Medicine, The Hospital for Sick Children, Canada, 555 University Ave., 2nd Floor, Atrium - Room 2830A, Toronto, ON, M5G 1X8, Canada
| | - Anne-Marie Guerguerian
- Institute of Biomaterials and Biomedical Engineering, University of Toronto, Rosebrugh Building (RS), 164 College Street, Room 407, Toronto, ON, M5S 3G9, Canada.,Department of Critical Care Medicine, The Hospital for Sick Children, Canada, 555 University Ave., 2nd Floor, Atrium - Room 2830A, Toronto, ON, M5G 1X8, Canada.,Neurosciences and Mental Health Research, The Hospital for Sick Children Research Institute, Peter Gilgan Centre for Research & Learning, 686 Bay Street, Toronto, ON, M5G 0A4, Canada
| | - Jessica Tomasi
- Institute of Biomaterials and Biomedical Engineering, University of Toronto, Rosebrugh Building (RS), 164 College Street, Room 407, Toronto, ON, M5S 3G9, Canada
| | - Peter Laussen
- Department of Critical Care Medicine, The Hospital for Sick Children, Canada, 555 University Ave., 2nd Floor, Atrium - Room 2830A, Toronto, ON, M5G 1X8, Canada
| | - Patricia Trbovich
- Institute of Biomaterials and Biomedical Engineering, University of Toronto, Rosebrugh Building (RS), 164 College Street, Room 407, Toronto, ON, M5S 3G9, Canada. .,Institute of Health Policy, Management and Evaluation, University of Toronto, 155 College St., Suite 425, Toronto, ON, M5T 3M6, Canada. .,Research and Innovation, North York General Hospital, 4001 Leslie Street, Toronto, ON, M2K 1E1, Canada.
| |
Collapse
|
18
|
Eye tracking to investigate cue processing in medical decision-making: A scoping review. COMPUTERS IN HUMAN BEHAVIOR 2017. [DOI: 10.1016/j.chb.2016.09.022] [Citation(s) in RCA: 32] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
19
|
Lund H. Eye tracking in library and information science: a literature review. LIBRARY HI TECH 2016. [DOI: 10.1108/lht-07-2016-0085] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Purpose
The purpose of this paper is to present a systematic literature review of the application of eye-tracking technology within the field of library and information science. Eye-tracking technology has now reached a level of maturity, which makes the use of the technology more accessible. Subsequently, a growing interest in employing eye tracking as a methodology within library and information science research must be anticipated.
Design/methodology/approach
The review follows the guidelines set in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses recommendations. Two reference databases are searched for relevant references: Library and Information Science Abstracts and Library, Information Science and Technology Abstracts. The main selection criteria are peer-reviewed literature that describes the experimental setting, including which eye-tracking equipment was used, the number of test persons and reports on the eye-tracking measures. Furthermore, this study will report which other methods were applied in combination with eye tracking.
Findings
The number of published research utilizing eye-tracking technologies within library and information science (LIS) is still limited although an increase in the use of eye-tracking technologies is observed during recent years.
Originality/value
To the knowledge of the author, this is the first systematic review on eye-tracking technology and application in LIS.
Collapse
|
20
|
Fong A, Hoffman DJ, Zachary Hettinger A, Fairbanks RJ, Bisantz AM. Identifying visual search patterns in eye gaze data; gaining insights into physician visual workflow. J Am Med Inform Assoc 2016; 23:1180-1184. [DOI: 10.1093/jamia/ocv196] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2015] [Revised: 09/28/2015] [Accepted: 11/10/2015] [Indexed: 11/12/2022] Open
Abstract
Abstract
Importance and Objectives As health information technologies become more prevalent in physician workflow, it is increasingly important to understand how physicians are using and interacting with these systems. This includes understanding how physicians search for information presented through health information technology systems. Eye tracking technologies provide a useful technique to understand how physicians visually search for information. However, analyzing eye tracking data can be challenging and is often done by measuring summative metrics, such as total time looking at a specific area and first-order transitions.
Methods In this paper, we propose an algorithmic approach to identify different visual search patterns. We demonstrate this approach by identifying common visual search patterns from physicians using a simulated prototype emergency department patient tracking system.
Results and Conclusions We evaluate and compare the visual search pattern results to first-order transition results. We discuss the benefits and limitations of this approach and insights from this initial evaluation.
Collapse
Affiliation(s)
- Allan Fong
- MedStar Institute for Innovation - National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, DC, 20008, USA
| | - Daniel J Hoffman
- MedStar Institute for Innovation - National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, DC, 20008, USA
| | - A Zachary Hettinger
- MedStar Institute for Innovation - National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, DC, 20008, USA
- Georgetown University School of Medicine, 3800 Reservoir Rd NW, Washington, DC 20007
| | - Rollin J Fairbanks
- MedStar Institute for Innovation - National Center for Human Factors in Healthcare, 3007 Tilden St. NW, Suite 7M, Washington, DC, 20008, USA
- Georgetown University School of Medicine, 3800 Reservoir Rd NW, Washington, DC 20007
- The State University of New York, University at Buffalo, Amherst, NY 14216, USA
| | - Ann M Bisantz
- The State University of New York, University at Buffalo, Amherst, NY 14216, USA
| |
Collapse
|
21
|
Badgeley MA, Shameer K, Glicksberg BS, Tomlinson MS, Levin MA, McCormick PJ, Kasarskis A, Reich DL, Dudley JT. EHDViz: clinical dashboard development using open-source technologies. BMJ Open 2016; 6:e010579. [PMID: 27013597 PMCID: PMC4809078 DOI: 10.1136/bmjopen-2015-010579] [Citation(s) in RCA: 42] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
OBJECTIVE To design, develop and prototype clinical dashboards to integrate high-frequency health and wellness data streams using interactive and real-time data visualisation and analytics modalities. MATERIALS AND METHODS We developed a clinical dashboard development framework called electronic healthcare data visualization (EHDViz) toolkit for generating web-based, real-time clinical dashboards for visualising heterogeneous biomedical, healthcare and wellness data. The EHDViz is an extensible toolkit that uses R packages for data management, normalisation and producing high-quality visualisations over the web using R/Shiny web server architecture. We have developed use cases to illustrate utility of EHDViz in different scenarios of clinical and wellness setting as a visualisation aid for improving healthcare delivery. RESULTS Using EHDViz, we prototyped clinical dashboards to demonstrate the contextual versatility of EHDViz toolkit. An outpatient cohort was used to visualise population health management tasks (n=14,221), and an inpatient cohort was used to visualise real-time acuity risk in a clinical unit (n=445), and a quantified-self example using wellness data from a fitness activity monitor worn by a single individual was also discussed (n-of-1). The back-end system retrieves relevant data from data source, populates the main panel of the application and integrates user-defined data features in real-time and renders output using modern web browsers. The visualisation elements can be customised using health features, disease names, procedure names or medical codes to populate the visualisations. The source code of EHDViz and various prototypes developed using EHDViz are available in the public domain at http://ehdviz.dudleylab.org. CONCLUSIONS Collaborative data visualisations, wellness trend predictions, risk estimation, proactive acuity status monitoring and knowledge of complex disease indicators are essential components of implementing data-driven precision medicine. As an open-source visualisation framework capable of integrating health assessment, EHDViz aims to be a valuable toolkit for rapid design, development and implementation of scalable clinical data visualisation dashboards.
Collapse
Affiliation(s)
- Marcus A Badgeley
- Harris Center for Precision Wellness, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
- Department of Genetics and Genomic Sciences, Icahn Institute for Genomics and Multiscale Biology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - Khader Shameer
- Harris Center for Precision Wellness, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
- Department of Genetics and Genomic Sciences, Icahn Institute for Genomics and Multiscale Biology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - Benjamin S Glicksberg
- Harris Center for Precision Wellness, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
- Department of Genetics and Genomic Sciences, Icahn Institute for Genomics and Multiscale Biology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - Max S Tomlinson
- Harris Center for Precision Wellness, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
- Department of Genetics and Genomic Sciences, Icahn Institute for Genomics and Multiscale Biology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - Matthew A Levin
- Department of Genetics and Genomic Sciences, Icahn Institute for Genomics and Multiscale Biology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
- Department of Anesthesiology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - Patrick J McCormick
- Department of Anesthesiology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - Andrew Kasarskis
- Department of Genetics and Genomic Sciences, Icahn Institute for Genomics and Multiscale Biology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - David L Reich
- Department of Anesthesiology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| | - Joel T Dudley
- Harris Center for Precision Wellness, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
- Department of Genetics and Genomic Sciences, Icahn Institute for Genomics and Multiscale Biology, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
- Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York City, New York, USA
| |
Collapse
|
22
|
Helldén A, Al-Aieshy F, Bastholm-Rahmner P, Bergman U, Gustafsson LL, Höök H, Sjöviker S, Söderström A, Odar-Cederlöf I. Development of a computerised decisions support system for renal risk drugs targeting primary healthcare. BMJ Open 2015; 5:e006775. [PMID: 26150141 PMCID: PMC4499680 DOI: 10.1136/bmjopen-2014-006775] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
OBJECTIVES To assess general practitioners (GPs) experience from the implementation and use of a renal computerised decision support system (CDSS) for drug dosing, developed for primary healthcare, integrated into the patient's electronic health record (EHR), and building on estimation of the patient's creatinine clearance (ClCG). DESIGN Qualitative research design by a questionnaire and a focus group discussion. SETTING AND PARTICIPANTS Eight GPs at two primary healthcare centres (PHCs). INTERVENTIONS The GP at PHC 1, and the project group, developed and tested the technical solution of the CDSS. Proof-of-concept was tested by seven GPs at PHC 2. They also participated in a group discussion and answered a questionnaire. A web window in the EHR gave drug and dosage in relation to ClCG. Each advice was according to three principles: If? Why? Because. OUTCOME MEASURES (1) The GPs' experience of 'easiness to use' and 'perceived usefulness' at PHC 2, based on loggings of use, answers from a questionnaire using a 5-point Likert scale, and answers from a focus group discussion. (2) The number of patients aged 65 years and older with an estimation of ClCG before and after the implementation of the CDSS. RESULTS The GPs found the CDSS fast, simple and easy to use. They appreciated the automatic presentation of the CICG status on opening the medication list, and the ability to actively look up specific drug recommendations in two steps. The CDSS scored high on the Likert scale. All GPs wanted to continue the use of the CDSS and to recommend it to others. The number of patients with an estimated ClCG increased 1.6-fold. CONCLUSIONS Acceptance of the simple graphical interface of this push and pull renal CDSS was high among the primary care physicians evaluating this proof of concept. The graphical model should be useful for further development of renal decision support systems.
Collapse
Affiliation(s)
- Anders Helldén
- Department of Clinical Pharmacology, Karolinska University Hospital, Stockholm, Sweden
- Division of Clinical Pharmacology, Department of Laboratory Medicine, Karolinska Institutet, Stockholm, Sweden
| | - Fadiea Al-Aieshy
- Department of Clinical Pharmacology, Karolinska University Hospital, Stockholm, Sweden
| | - Pia Bastholm-Rahmner
- Division of Clinical Pharmacology, Department of Laboratory Medicine, Karolinska Institutet, Stockholm, Sweden
- Department of Healthcare Development, Public Healthcare Administration, Stockholm County Council, Stockholm, Sweden
| | - Ulf Bergman
- Department of Clinical Pharmacology, Karolinska University Hospital, Stockholm, Sweden
- Division of Clinical Pharmacology, Department of Laboratory Medicine, Karolinska Institutet, Stockholm, Sweden
| | - Lars L Gustafsson
- Department of Clinical Pharmacology, Karolinska University Hospital, Stockholm, Sweden
- Division of Clinical Pharmacology, Department of Laboratory Medicine, Karolinska Institutet, Stockholm, Sweden
| | - Hans Höök
- Department of Healthcare Development, Public Healthcare Administration, Stockholm County Council, Stockholm, Sweden
| | - Susanne Sjöviker
- Department of Healthcare Development, Public Healthcare Administration, Stockholm County Council, Stockholm, Sweden
| | | | - Ingegerd Odar-Cederlöf
- Department of Clinical Pharmacology, Karolinska University Hospital, Stockholm, Sweden
- Division of Clinical Pharmacology, Department of Laboratory Medicine, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
23
|
Asan O, Yang Y. Using Eye Trackers for Usability Evaluation of Health Information Technology: A Systematic Literature Review. JMIR Hum Factors 2015; 2:e5. [PMID: 27026079 PMCID: PMC4797658 DOI: 10.2196/humanfactors.4062] [Citation(s) in RCA: 30] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2014] [Revised: 12/23/2014] [Accepted: 03/01/2015] [Indexed: 11/17/2022] Open
Abstract
Background Eye-tracking technology has been used to measure human cognitive processes and has the potential to improve the usability of health information technology (HIT). However, it is still unclear how the eye-tracking method can be integrated with other traditional usability methodologies to achieve its full potential. Objective The objective of this study was to report on HIT evaluation studies that have used eye-tracker technology, and to envision the potential use of eye-tracking technology in future research. Methods We used four reference databases to initially identify 5248 related papers, which resulted in only 9 articles that met our inclusion criteria. Results Eye-tracking technology was useful in finding usability problems in many ways, but is still in its infancy for HIT usability evaluation. Limited types of HITs have been evaluated by eye trackers, and there has been a lack of evaluation research in natural settings. Conclusions More research should be done in natural settings to discover the real contextual-based usability problems of clinical and mobile HITs using eye-tracking technology with more standardized methodologies and guidance.
Collapse
Affiliation(s)
- Onur Asan
- Center for Patient Care and Outcomes Research, Department of Medicine, Medical College of Wisconsin, Milwaukee, WI, United States.
| | | |
Collapse
|