1
|
Thomas A, Asnes A, Libby K, Hsiao A, Tiyyagura G. Developing and Testing the Usability of a Novel Child Abuse Clinical Decision Support System: Mixed Methods Study. J Med Internet Res 2024; 26:e51058. [PMID: 38551639 PMCID: PMC11015363 DOI: 10.2196/51058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 02/23/2024] [Accepted: 02/23/2024] [Indexed: 04/02/2024] Open
Abstract
BACKGROUND Despite the impact of physical abuse on children, it is often underdiagnosed, especially among children evaluated in emergency departments (EDs). Electronic clinical decision support (CDS) can improve the recognition of child physical abuse. OBJECTIVE We aimed to develop and test the usability of a natural language processing-based child abuse CDS system, known as the Child Abuse Clinical Decision Support (CA-CDS), to alert ED clinicians about high-risk injuries suggestive of abuse in infants' charts. METHODS Informed by available evidence, a multidisciplinary team, including an expert in user design, developed the CA-CDS prototype that provided evidence-based recommendations for the evaluation and management of suspected child abuse when triggered by documentation of a high-risk injury. Content was customized for medical versus nursing providers and initial versus subsequent exposure to the alert. To assess the usability of and refine the CA-CDS, we interviewed 24 clinicians from 4 EDs about their interactions with the prototype. Interview transcripts were coded and analyzed using conventional content analysis. RESULTS Overall, 5 main categories of themes emerged from the study. CA-CDS benefits included providing an extra layer of protection, providing evidence-based recommendations, and alerting the entire clinical ED team. The user-centered, workflow-compatible design included soft-stop alert configuration, editable and automatic documentation, and attention-grabbing formatting. Recommendations for improvement included consolidating content, clearer design elements, and adding a hyperlink with additional resources. Barriers to future implementation included alert fatigue, hesitancy to change, and concerns regarding documentation. Facilitators of future implementation included stakeholder buy-in, provider education, and sharing the test characteristics. On the basis of user feedback, iterative modifications were made to the prototype. CONCLUSIONS With its user-centered design and evidence-based content, the CA-CDS can aid providers in the real-time recognition and evaluation of infant physical abuse and has the potential to reduce the number of missed cases.
Collapse
Affiliation(s)
- Amy Thomas
- Department of Pediatrics, Yale University School of Medicine, New Haven, CT, United States
| | - Andrea Asnes
- Department of Pediatrics, Yale University School of Medicine, New Haven, CT, United States
| | - Kyle Libby
- 3M | M*Modal, 3M Health Information Systems, 3M Company, Maplewood, MN, United States
| | - Allen Hsiao
- Department of Pediatrics, Yale University School of Medicine, New Haven, CT, United States
| | - Gunjan Tiyyagura
- Department of Pediatrics, Yale University School of Medicine, New Haven, CT, United States
| |
Collapse
|
2
|
Blecker S, Gannon M, De Leon S, Shelley D, Wu WY, Tabaei B, Magno J, Pham-Singer H. Practice facilitation for scale up of clinical decision support for hypertension management: study protocol for a cluster randomized control trial. Contemp Clin Trials 2023; 129:107177. [PMID: 37037392 PMCID: PMC10871131 DOI: 10.1016/j.cct.2023.107177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Revised: 03/09/2023] [Accepted: 04/04/2023] [Indexed: 04/12/2023]
Abstract
BACKGROUND Only half of patients with hypertension have adequately controlled blood pressure. Clinical decision support (CDS) has the potential to overcome barriers to delivering guideline-recommended care and improve hypertension management. However, optimal strategies for scaling CDS have not been well established, particularly in small, independent primary care practices which often lack the resources to effectively change practice routines. Practice facilitation is an implementation strategy that has been shown to support process changes. Our objective is to evaluate whether practice facilitation provided with hypertension-focused CDS can lead to improvements in blood pressure control for patients seen in small primary care practices. METHODS/DESIGN We will conduct a cluster randomized control trial to compare the effect of hypertension-focused CDS plus practice facilitation on BP control, as compared to CDS alone. The practice facilitation intervention will include an initial training in the CDS and a review of current guidelines along with follow-up for coaching and integration support. We will randomize 46 small primary care practices in New York City who use the same electronic health record vendor to intervention or control. All patients with hypertension seen at these practices will be included in the evaluation. We will also assess implementation of CDS in all practices and practice facilitation in the intervention group. DISCUSSION The results of this study will inform optimal implementation of CDS into small primary care practices, where much of care delivery occurs in the U.S. Additionally, our assessment of barriers and facilitators to implementation will support future scaling of the intervention. CLINICALTRIALS gov Identifier: NCT05588466.
Collapse
Affiliation(s)
- Saul Blecker
- NYU Grossman School of Medicine, New York, NY, United States of America.
| | - Matthew Gannon
- New York City Department of Health and Mental Hygiene, New York, NY, United States of America
| | - Samantha De Leon
- New York City Department of Health and Mental Hygiene, New York, NY, United States of America
| | - Donna Shelley
- NYU School of Global Public Health, New York, NY, United States of America
| | - Winfred Y Wu
- University of Miami - Miller School of Medicine, Miami, FL, United States of America
| | - Bahman Tabaei
- New York City Department of Health and Mental Hygiene, New York, NY, United States of America
| | - Janice Magno
- New York City Department of Health and Mental Hygiene, New York, NY, United States of America
| | - Hang Pham-Singer
- New York City Department of Health and Mental Hygiene, New York, NY, United States of America
| |
Collapse
|
3
|
Feldman J, Goodman A, Hochman K, Chakravartty E, Austrian J, Iturrate E, Bosworth B, Saxena A, Moussa MM, Chenouda DM, Volpicelli F, Adler N, Weisstuch J, Testa P. Novel Note Templates to Enhance Signal and Reduce Noise in Medical Documentation: a Prospective Improvement Study. JMIR Form Res 2023; 7:e41223. [PMID: 36821760 PMCID: PMC10134024 DOI: 10.2196/41223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 01/23/2023] [Accepted: 02/15/2023] [Indexed: 02/17/2023] Open
Abstract
BACKGROUND The introduction of electronic workflows has allowed for the flow of raw un-contextualized clinical data into medical documentation. As a result, many electronic notes have become replete of "noise" and deplete of clinically significant "signals". There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating. OBJECTIVE To describe the development and impact of a novel set of templates designed to change the flow of information in medical documentation. METHODS This is a multi-hospital nonrandomized prospective improvement study conducted on the Inpatient General Internal Medicine Service across three hospital campuses at the New York University (NYU) Langone Health System. A group of physician leaders representing each campus met biweekly for six months. The output of these meetings included 1) a conceptualization of the note bloat problem as a dysfunction in information flow 2) a set of guiding principles for organizational documentation improvement 3) the design and build of novel electronic templates that reduced the flow of extraneous information into provider notes by providing link outs to best practice data visualizations and 4) a documentation improvement curriculum for inpatient medicine providers. Prior to go-live, pragmatic usability testing was performed with the new progress note template, and the overall user experience measured using the System Usability Scale (SUS). Primary outcomes measures after go-live include template utilization rate and note length in characters. RESULTS In usability testing amongst 22 medicine providers, the new progress note template averaged a usability score of 90.6/100 on the System Usability Scale. 77% of providers strongly agreed that the new template was easy to use. 68% strongly agreed that they would like to use the template frequently. In the three months after template implementation, General Internal Medicine providers wrote 65% of all inpatient notes with the new templates. During this period of time the organization saw a 46%, 47%, and 32% reduction in note length for general medicine progress notes, consults, and H&Ps, respectively, when compared to a baseline measurement period prior to interventions. CONCLUSIONS A bundled intervention that included deployment of novel templates for inpatient general medicine providers significantly reduced average note length on the clinical service. Templates designed to reduce the flow of extraneous information into provider notes performed well during usability testing, and these templates were rapidly adopted across all hospital campuses. Further research is needed to assess the impact of novel templates on note quality, provider efficiency and patient outcomes. CLINICALTRIAL
Collapse
Affiliation(s)
- Jonah Feldman
- Medical Center Information Technology, NYU Langone Health, New York, US.,Department of Medicine, NYU Long Island School of Medicine, Mineola, US
| | - Adam Goodman
- Division of Gastroenterology & Hepatology, NYU Grossman School of Medicine, New York,, US
| | - Katherine Hochman
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Eesha Chakravartty
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US.,Medical Center Information Technology, NYU Langone Health, New York, US
| | - Jonathan Austrian
- Medical Center Information Technology, NYU Langone Health, New York, US.,Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Eduardo Iturrate
- Medical Center Information Technology, NYU Langone Health, New York, US.,Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Brian Bosworth
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Archana Saxena
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Marwa M Moussa
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Dina M Chenouda
- Department of Medicine, NYU Long Island School of Medicine, Mineola, US
| | | | - Nicole Adler
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | | | - Paul Testa
- Medical Center Information Technology, NYU Langone Health, New York, US
| |
Collapse
|
4
|
Ogundipe A, Sim TF, Emmerton L. Health information communication technology evaluation frameworks for pharmacist prescribing: A systematic scoping review. Res Social Adm Pharm 2023; 19:218-234. [PMID: 36220754 DOI: 10.1016/j.sapharm.2022.09.010] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Revised: 09/07/2022] [Accepted: 09/18/2022] [Indexed: 01/25/2023]
Abstract
BACKGROUND Information communication technology (ICT) is instrumental in pharmacists' current practice and emerging roles. One such role is prescribing, which requires the use of clinical guidelines and documentation of decision-making, commonly via ICT. Development and refinement of ICT should be guided by evaluation frameworks that describe or measure features of ICT and its implementation. In the context of pharmacist prescribing, these evaluation frameworks should be specific to health stakeholders and the pharmacy setting. OBJECTIVES To identify ICT evaluation frameworks from health-related literature and review frameworks relevant to the development, implementation, and evaluation of pharmacist prescribing. METHODS A database search of CINAHL, Cochrane Library, EMBASE, Medline (Ovid), ProQuest, Scopus, Web of Science and grey literature was conducted, using combinations of keywords relating to 'ICT', 'utilization', 'usability', and 'evaluation framework'. Abstracts and titles were screened according to inclusion criteria. Identified evaluation frameworks were critiqued for relevance to pharmacy practice. RESULTS Twenty-two articles were identified, describing the development or application of 20 evaluation frameworks. None of the frameworks was developed specifically for pharmacy practice. The Technology Acceptance Model (TAM), describing use behavior, behavior intention, perceived usefulness, and perceived ease of use, was the most widely utilized framework. The Information System Success (ISS) and Human-Organization and Technology Fit (HOT-fit) are notable evaluation frameworks that address user and organizational influences in health ICT utility, and factors of both can address the limitation of TAM. CONCLUSIONS The findings call for development of an agile evaluation framework for the system under review; however, this can prove difficult due to the heterogenicity and complexity of the healthcare system, particularly contemporary pharmacy practice. While the TAM appears useful to evaluate user attitudes and intentions towards ICT, its relevance to ICT in contemporary community pharmacy practice requires exploration.
Collapse
Affiliation(s)
- Ayomide Ogundipe
- Curtin Medical School, Curtin University, Kent Street, 6102, Western Australia, Australia.
| | - Tin Fei Sim
- Curtin Medical School, Curtin University, Kent Street, 6102, Western Australia, Australia
| | - Lynne Emmerton
- Curtin Medical School, Curtin University, Kent Street, 6102, Western Australia, Australia
| |
Collapse
|
5
|
Herrijgers C, Platteau T, Vandebosch H, Poels K, Florence E. Using Intervention Mapping to Develop an mHealth Intervention to Support Men Who Have Sex With Men Engaging in Chemsex (Budd): Development and Usability Study. JMIR Res Protoc 2022; 11:e39678. [PMID: 36542451 DOI: 10.2196/39678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2022] [Revised: 08/25/2022] [Accepted: 10/25/2022] [Indexed: 12/24/2022] Open
Abstract
BACKGROUND Chemsex refers to the intentional use of drugs before or during sex among men who have sex with men (MSM). Engaging in chemsex has been linked to significant negative impacts on physical, psychological, and social well-being. However, no evidence-based support tools have addressed either these harms or the care needs of MSM who engage in chemsex. OBJECTIVE The purpose of this paper was to describe the development of a mobile health intervention (named Budd) using the intervention mapping protocol (IMP). Budd aims to support and inform MSM who participate in chemsex, reduce the negative impacts associated with chemsex, and encourage more reasoned participation. METHODS The IMP consists of 6 steps to develop, implement, and evaluate evidence-based health interventions. A needs assessment was carried out between September 2, 2019, and March 31, 2020, by conducting a literature study and in-depth interviews. Change objectives were selected based on these findings, after which theory-based intervention methods were selected. The first version of the intervention was developed in December 2020 and pilot-tested between February 1, 2021, and April 30, 2021. Adjustments were made based on the findings from this study. A separate article will be dedicated to the effectiveness study, conducted between October 15, 2021, and February 24, 2022, and implementation of the intervention. The Budd app went live in April 2022. RESULTS Budd aims to address individual factors and support chemsex participants in applying harm reduction measures when taking drugs (drug information, drug combination tool, and notebook), preparing for participation in a chemsex session (articles on chemsex, preparation tool, and event-specific checklist), planning sufficient time after a chemsex session to recover (planning tool), seeking support for their chemsex participation (overview of existing local health care and peer support services, reflection, personal statistics, and user testimonials), taking HIV medication or pre-exposure prophylaxis in a timely manner during a chemsex session (preparation tool), and contacting emergency services in case of an emergency and giving first aid to others (emergency information and personal buddy). CONCLUSIONS The IMP proved to be a valuable tool in the planning and development of the Budd app. This study provides researchers and practitioners with valuable information that may help them to set up their own health interventions. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) RR1-10.2196/39678.
Collapse
Affiliation(s)
- Corinne Herrijgers
- Department of Clinical Sciences, Institute of Tropical Medicine, Antwerpen, Belgium
| | - Tom Platteau
- Department of Clinical Sciences, Institute of Tropical Medicine, Antwerpen, Belgium
| | - Heidi Vandebosch
- Department of Communication Studies, University of Antwerp, Antwerpen, Belgium
| | - Karolien Poels
- Department of Communication Studies, University of Antwerp, Antwerpen, Belgium
| | - Eric Florence
- Department of Clinical Sciences, Institute of Tropical Medicine, Antwerpen, Belgium
| |
Collapse
|
6
|
Wiberg AC, Ghaderi A, Danielsson HB, Safarzadeh K, Parling T, Carlbring P, Jansson M, Welch E. Internet-based cognitive behavior therapy for eating disorders - Development and feasibility evaluation. Internet Interv 2022; 30:100570. [PMID: 36110307 PMCID: PMC9468502 DOI: 10.1016/j.invent.2022.100570] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Revised: 08/20/2022] [Accepted: 08/29/2022] [Indexed: 11/28/2022] Open
Abstract
BACKGROUND Eating disorders (ED) are severe psychiatric conditions, characterized by decreased quality of life and high mortality. However, only a minority of patients with ED seek care and very few receive treatment. Internet-delivered cognitive behavioral therapy (ICBT) has the potential to increase access to evidence-based treatments. AIMS The aims of the present study were to (1) develop and evaluate the usability of an Internet-delivered guided self-help treatment based on Enhanced Cognitive Behavioral Therapy (ICBT-E) for patients with full or subthreshold bulimia nervosa (BN) or binge eating disorder (BED) with a user centered design process, and (2) to evaluate its feasibility and preliminary outcome in a clinical environment. METHOD The study was undertaken in two stages. In Stage I, a user-centered design approach was applied with iterative phases of prototype development and evaluation. Participants were eight clinicians and 30 individuals with current or previous history of ED. In Stage II, 41 patients with full or subthreshold BN or BED were recruited to a single-group open trial to evaluate the feasibility and preliminary outcome of ICBT-E. Primary outcome variables were diagnostic status and self-rated ED symptoms. RESULTS The user-centered design process was instrumental in the development of the ICBT-E, by contributing to improvements of the program and to the content being adapted to the needs and preferences of end-users. The overall usability of the program was found to be good. ICBT-E targets key maintaining factors in ED by introducing healthy eating patterns and addressing over-evaluation of weight and shape. The results indicate that ICBT-E, delivered in a clinical setting, is a feasible and promising treatment for full or subthreshold BN or BED, with a high level of acceptability observed and treatment completion of 73.2 %. Participation in ICBT-E was associated with significant symptom reductions in core ED symptomology, functional impairment as well as depressive symptoms, and the results were maintained at the 3-month follow-up. CONCLUSIONS ICBT-E was developed with end-users' preferences in mind, in accordance with the identified recommendations, and the program was perceived as usable by end-users. The study demonstrated the potential of ICBT-E, which marks a step forward in the effort to make powerful, empirically supported psychological interventions targeting ED more widely available and accessible.
Collapse
Affiliation(s)
- Anne-Charlotte Wiberg
- Centre for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm Health Care Services, Stockholm County Council, Stockholm, Sweden
| | - Ata Ghaderi
- Department of Clinical Neuroscience, Karolinska Institutet, Nobels väg 9, 17177 Stockholm, Sweden
| | | | - Kousha Safarzadeh
- Student Health Center, Lund University, Sandgatan 3, 22350 Lund, Sweden
| | - Thomas Parling
- Centre for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm Health Care Services, Stockholm County Council, Stockholm, Sweden
| | - Per Carlbring
- Department of Psychology, Stockholm University, 106 91 Stockholm, Sweden
| | - Magdalena Jansson
- Stockholm Center for Eating Disorders, Stockholm County Council, Wollmar Yxkullsgatan 27B, 118 50 Stockholm, Sweden
| | - Elisabeth Welch
- Centre for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm Health Care Services, Stockholm County Council, Stockholm, Sweden
| |
Collapse
|
7
|
Tossaint-Schoenmakers R, Kasteleyn MJ, Rauwerdink A, Chavannes N, Willems S, Talboom-Kamp EPWA. Development of a quality management model and self-assessment questionnaire for hybrid health care: a concept mapping study (Preprint). JMIR Form Res 2022; 6:e38683. [PMID: 35797097 PMCID: PMC9305399 DOI: 10.2196/38683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Revised: 06/01/2022] [Accepted: 06/07/2022] [Indexed: 11/13/2022] Open
Affiliation(s)
- Rosian Tossaint-Schoenmakers
- Saltro Diagnostic Centre, Unilabs Netherlands, Utrecht, Netherlands
- National eHealth Living Lab, Leiden University Medical Centre, Leiden, Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands
| | - Marise J Kasteleyn
- National eHealth Living Lab, Leiden University Medical Centre, Leiden, Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands
| | - Anneloek Rauwerdink
- Department of Surgery, Gastroenterology and Metabolism, Amsterdam University Medical Centre, Amsterdam, Netherlands
| | - Niels Chavannes
- National eHealth Living Lab, Leiden University Medical Centre, Leiden, Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands
| | - Sofie Willems
- National eHealth Living Lab, Leiden University Medical Centre, Leiden, Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands
| | - Esther P W A Talboom-Kamp
- National eHealth Living Lab, Leiden University Medical Centre, Leiden, Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands
- Unilabs Group, Geneve, Switzerland
| |
Collapse
|
8
|
Awoonor‐Williams JK, Phillips JF. Developing organizational learning for scaling-up community-based primary health care in Ghana. Learn Health Syst 2022; 6:e10282. [PMID: 35036554 PMCID: PMC8753302 DOI: 10.1002/lrh2.10282] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2020] [Revised: 06/01/2021] [Accepted: 06/03/2021] [Indexed: 11/09/2022] Open
Abstract
INTRODUCTION Achieving effective community-based primary health care requires evidence for guiding strategic decisions that must be made. However, research processes often limit data collection to particular organizational levels or disseminate results to specific audiences. Decision-making that emerges can fail to account for the contrasting perspectives and needs of managers at each organizational level. The Ghana Health Service (GHS) addressed this problem with a multilevel and sequential research and action approach that has provided two decades of implementation learning for guiding community-based primary health care development. METHOD The GHS implementation research initiatives progressed from (i) a participatory pilot investigation to (ii) an experimental trial of strategies that emerged to (iii) replication research for testing scale-up, culminating in (iv) evidence-based scale-up of a national community-based primary health care program. A reform process subsequently repeated this sequence in a manner that involved stakeholders at the community, sub-district, district, and regional levels of the system. The conduct, interpretation, and dissemination of results that emerged comprised a strategy for achieving systems learning by conducting investigations in phases in conjunction with bottom-up knowledge capture, lateral exchanges for fostering peer learning at each system level, and top-down processes for communicating results as policy. Continuous accumulation of qualitative data on stakeholder reactions to operations at each organizational level was conducted in conjunction with quantitative monitoring of field operations. RESULTS Implementation policies were enhanced by results associated with each phase. A quasi-experiment for testing the reform process showed that scale-up of community-based primary health care was accelerated, leading to improvements in childhood survival and reduced fertility. CONCLUSION Challenges to system learning were overcome despite severe resource constraints. The integration of knowledge generation with ongoing management processes institutionalized learning for achieving evidence-driven program action.
Collapse
Affiliation(s)
| | - James F. Phillips
- Heilbrunn Department of Population and Family Health, Mailman School of Public HealthColumbia UniversityNew YorkNew YorkUSA
| |
Collapse
|
9
|
Austrian J, Mendoza F, Szerencsy A, Fenelon L, Horwitz LI, Jones S, Kuznetsova M, Mann DM. Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials. J Med Internet Res 2021; 23:e16651. [PMID: 33835035 PMCID: PMC8065554 DOI: 10.2196/16651] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2019] [Revised: 08/14/2020] [Accepted: 03/11/2021] [Indexed: 12/21/2022] Open
Abstract
BACKGROUND Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools. OBJECTIVE This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care. METHODS A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior. RESULTS To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images. CONCLUSIONS These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS. TRIAL REGISTRATION Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.
Collapse
Affiliation(s)
- Jonathan Austrian
- Department of Medicine, NYU Grossman School of Medicine, New York, NY, United States.,Medical Center Information Technology, NYU Langone Health, New York, NY, United States
| | - Felicia Mendoza
- Department of Population Health, NYU Grossman School of Medicine, New York, NY, United States
| | - Adam Szerencsy
- Department of Medicine, NYU Grossman School of Medicine, New York, NY, United States.,Medical Center Information Technology, NYU Langone Health, New York, NY, United States
| | - Lucille Fenelon
- Medical Center Information Technology, NYU Langone Health, New York, NY, United States
| | - Leora I Horwitz
- Department of Medicine, NYU Grossman School of Medicine, New York, NY, United States.,Department of Population Health, NYU Grossman School of Medicine, New York, NY, United States
| | - Simon Jones
- Department of Population Health, NYU Grossman School of Medicine, New York, NY, United States
| | - Masha Kuznetsova
- Department of Population Health, NYU Grossman School of Medicine, New York, NY, United States
| | - Devin M Mann
- Department of Medicine, NYU Grossman School of Medicine, New York, NY, United States.,Medical Center Information Technology, NYU Langone Health, New York, NY, United States.,Department of Population Health, NYU Grossman School of Medicine, New York, NY, United States
| |
Collapse
|
10
|
How to Enhance Digital Support for Cross-Organisational Health Care Teams? A User-Based Explorative Study. JOURNAL OF HEALTHCARE ENGINEERING 2020; 2020:8824882. [PMID: 33029336 PMCID: PMC7528149 DOI: 10.1155/2020/8824882] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/05/2020] [Revised: 09/03/2020] [Accepted: 09/06/2020] [Indexed: 11/17/2022]
Abstract
Health care service provision of individualised treatment to an ageing population prone to chronic conditions and multimorbidities is threatened. There is a need for digitally supported care, that is, (1) person-centred, (2) integrated, and (3) proactive. The research project 3P, Patients and Professionals in Productive Teams, aimed to validate and verify the prerequisites for health care systems run with patient-centred service models. This paper presents an explorative study of the digital support of a cross-organisational health care team in Norway, providing services to elderly frail people with multimorbidities in hospital discharge transition. Qualitative research methods were employed, with interviews and observations to map and evaluate the information flow and the digital support of collaborative work across organisations. The evaluation showed a lacking interoperability between the digital systems and a limited support for cross-organisational teamwork, causing raised manual efforts to maintain the information flow. Tools for coordination and planning across organisations were lacking. To enhance the situation, principles for a cloud-based health portal are proposed with a shared workspace, teamwork functionality for cross-organisational health care teams, and automatic back-end synchronisation of stored information. The main implications of this paper lie in the proposed principles which are transferable to a multitude of clinical contexts, where ad-hoc based access to shared medical information is of importance for decision-making and life-saving treatment.
Collapse
|
11
|
Sigle S, Barriga P, Correa Fernández FJ, Juhra C, Härtel S, Fegeler C. Evaluating Online Consumer Medication Information Systems: Comparative Online Usability Study. JMIR Mhealth Uhealth 2020; 8:e16648. [PMID: 32490848 PMCID: PMC7301258 DOI: 10.2196/16648] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2019] [Revised: 02/14/2020] [Accepted: 03/11/2020] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND Medication is the most common intervention in health care, and the number of online consumer information systems within the pharmaceutical sector is increasing. However, online consumer information systems can be a barrier for users, imposing information asymmetries between stakeholders. OBJECTIVE The objective of this study was to quantify and compare the usability of an online consumer medication information system (OCMIS) against a reference implementation based on an interoperable information model for patients, physicians, and pharmacists. METHODS Quantitative and qualitative data were acquired from patients, physicians, and pharmacists in this online usability study. We administered 3 use cases and a post hoc questionnaire per user. Quantitative usability data including effectiveness (task success), efficiency (task time), and user satisfaction (system usability scale [SUS]) was complemented by qualitative and demographic data. Users evaluated 6 existing systems and 1 reference implementation of an OCMIS. RESULTS A total of 137 patients, 81 physicians, and 68 pharmacists participated in this study. Task success varied from 84% to 92% in patients, 66% to 100% in physicians, and 50% to 91% in pharmacists. Task completion time decreased over the course of the study for all but 2 OCMIS within the patient group. Due to an assumed nonnormal distribution of SUS scores, within-group comparison was done using the Kruskal-Wallis test. Patients showed differences in SUS scores (P=.02) and task time (P=.03), while physicians did not have significant differences in SUS scores (P=.83) and task time (P=.72). For pharmacists, a significant difference in SUS scores (P<.001) and task time (P=.007) was detected. CONCLUSIONS The vendor-neutral reference implementation based on an interoperable information model was proven to be a promising approach that was not inferior to existing solutions for patients and physicians. For pharmacists, it exceeded user satisfaction scores compared to other OCMIS. This data-driven approach based on an interoperable information model enables the development of more user-tailored features to increase usability. This fosters data democratization and empowers stakeholders within the pharmaceutical sector.
Collapse
Affiliation(s)
- Stefan Sigle
- Department of Telemedicine, University Clinic Münster, University of Münster, Münster, Germany.,Center for Medical Informatics and Telemedicine, Faculty of Medicine, University of Chile, Santiago, Chile.,MOLIT Institute, Heilbronn, Germany
| | - Pilar Barriga
- Center for Medical Informatics and Telemedicine, Faculty of Medicine, University of Chile, Santiago, Chile
| | | | - Christian Juhra
- Department of Telemedicine, University Clinic Münster, University of Münster, Münster, Germany
| | - Steffen Härtel
- Center for Medical Informatics and Telemedicine, Faculty of Medicine, University of Chile, Santiago, Chile
| | - Christian Fegeler
- MOLIT Institute, Heilbronn, Germany.,Institute for Medicine, Informatics and Economy, Faculty of Informatics, University of Heilbronn, Heilbronn, Germany
| |
Collapse
|
12
|
Ray JM, Ahmed OM, Solad Y, Maleska M, Martel S, Jeffery MM, Platts-Mills TF, Hess EP, D'Onofrio G, Melnick ER. Computerized Clinical Decision Support System for Emergency Department-Initiated Buprenorphine for Opioid Use Disorder: User-Centered Design. JMIR Hum Factors 2019; 6:e13121. [PMID: 30810531 PMCID: PMC6414819 DOI: 10.2196/13121] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2018] [Revised: 02/01/2019] [Accepted: 02/09/2019] [Indexed: 01/16/2023] Open
Abstract
BACKGROUND Emergency departments (EDs) frequently care for individuals with opioid use disorder (OUD). Buprenorphine (BUP) is an effective treatment option for patients with OUD that can safely be initiated in the ED. At present, BUP is rarely initiated as a part of routine ED care. Clinical decision support (CDS) could accelerate adoption of ED-initiated BUP into routine emergency care. OBJECTIVE This study aimed to design and formatively evaluate a user-centered decision support tool for ED initiation of BUP for patients with OUD. METHODS User-centered design with iterative prototype development was used. Initial observations and interviews identified workflows and information needs. The design team and key stakeholders reviewed prototype designs to ensure accuracy. A total of 5 prototypes were evaluated and iteratively refined based on input from 26 attending and resident physicians. RESULTS Early feedback identified concerns with the initial CDS design: an alert with several screens. The timing of the alert led to quick dismissal without using the tool. User feedback on subsequent iterations informed the development of a flexible tool to support clinicians with varied levels of experience with the intervention by providing both one-click options for direct activation of care pathways and user-activated support for critical decision points. The final design resolved challenging navigation issues through targeted placement, color, and design of the decision support modules and care pathways. In final testing, users expressed that the tool could be easily learned without training and was reasonable for use during routine emergency care. CONCLUSIONS A user-centered design process helped designers to better understand users' needs for a Web-based clinical decision tool to support ED initiation of BUP for OUD. The process identified varying needs across user experience and familiarity with the protocol, leading to a flexible design supporting both direct care pathways and user-initiated decision support.
Collapse
Affiliation(s)
- Jessica M Ray
- Yale School of Medicine, New Haven, CT, United States
| | - Osama M Ahmed
- Yale School of Medicine, New Haven, CT, United States
| | | | | | - Shara Martel
- Yale School of Medicine, New Haven, CT, United States
| | | | | | - Erik P Hess
- University of Alabama at Birmingham School of Medicine, Birmingham, AL, United States
| | | | | |
Collapse
|
13
|
Chokshi SK, Mann DM. Innovating From Within: A Process Model for User-Centered Digital Development in Academic Medical Centers. JMIR Hum Factors 2018; 5:e11048. [PMID: 30567688 PMCID: PMC6315266 DOI: 10.2196/11048] [Citation(s) in RCA: 28] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2018] [Revised: 10/11/2018] [Accepted: 10/12/2018] [Indexed: 01/23/2023] Open
Abstract
Background Design thinking and human-centered design approaches have become increasingly common in health care literature, particularly in relation to health information technology (HIT), as a pathway toward the development of usable, diffusible tools and processes. There is a need in academic medical centers tasked with digital innovation for a comprehensive process model to guide development that incorporates current industry trends, including design thinking and lean and agile approaches to digital development. Objective This study aims to describe the foundations and phases of our model for user-centered HIT development. Methods Based on our experience, we established an integrated approach and rigorous process for HIT development that leverages design thinking and lean and agile strategies in a pragmatic way while preserving methodological integrity in support of academic research goals. Results A four-phased pragmatic process model was developed for user-centered digital development in HIT. Conclusions The model for user-centered HIT development that we developed is the culmination of diverse innovation projects and represents a multiphased, high-fidelity process for making more creative, flexible, efficient, and effective tools. This model is a critical step in building a rigorous approach to HIT design that incorporates a multidisciplinary, pragmatic perspective combined with academic research practices and state-of-the-art approaches to digital product development to meet the unique needs of health care.
Collapse
Affiliation(s)
- Sara Kuppin Chokshi
- Department of Population Health, New York University School of Medicine, New York, NY, United States
| | - Devin M Mann
- Department of Population Health, New York University School of Medicine, New York, NY, United States.,New York Univeristy Langone Health, Medical Center Information Technology, New York, NY, United States
| |
Collapse
|