1
|
Weiner M, Flanagan ME, Ernst K, Cottingham AH, Rattray NA, Franks Z, Savoy AW, Lee JL, Frankel RM. Accuracy, thoroughness, and quality of outpatient primary care documentation in the U.S. Department of Veterans Affairs. BMC PRIMARY CARE 2024; 25:262. [PMID: 39026167 PMCID: PMC11264844 DOI: 10.1186/s12875-024-02501-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/07/2024] [Accepted: 06/27/2024] [Indexed: 07/20/2024]
Abstract
BACKGROUND Electronic health records (EHRs) can accelerate documentation and may enhance details of notes, or complicate documentation and introduce errors. Comprehensive assessment of documentation quality requires comparing documentation to what transpires during the clinical encounter itself. We assessed outpatient primary care notes and corresponding recorded encounters to determine accuracy, thoroughness, and several additional key measures of documentation quality. METHODS Patients and primary care clinicians across five midwestern primary care clinics of the US Department of Veterans Affairs were recruited into a prospective observational study. Clinical encounters were video-recorded and transcribed verbatim. Using the Physician Documentation Quality Instrument (PDQI-9) added to other measures, reviewers scored quality of the documentation by comparing transcripts to corresponding encounter notes. PDQI-9 items were scored from 1 to 5, with higher scores indicating higher quality. RESULTS Encounters (N = 49) among 11 clinicians were analyzed. Most issues that patients initiated in discussion were omitted from notes, and nearly half of notes referred to information or observations that could not be verified. Four notes lacked concluding assessments and plans; nine lacked information about when patients should return. Except for thoroughness, PDQI-9 items that were assessed achieved quality scores exceeding 4 of 5 points. CONCLUSIONS Among outpatient primary care electronic records examined, most issues that patients initiated in discussion were absent from notes, and nearly half of notes referred to information or observations absent from transcripts. EHRs may contribute to certain kinds of errors. Approaches to improving documentation should consider the roles of the EHR, patient, and clinician together.
Collapse
Affiliation(s)
- Michael Weiner
- Center for Health Information and Communication, Department of Veterans Affairs, Health Services Research and Development Service, Veterans Health Administration, Richard L. Roudebush VA Medical Center, CIN 13 416, Indianapolis, IN, USA.
- Department of Medicine, Indiana University School of Medicine, Indianapolis, IN, USA.
- Indiana University Center for Health Services and Outcomes Research, Regenstrief Institute, Inc, 1101 West 10th Street, 46202-4800, Indianapolis, IN, USA.
| | - Mindy E Flanagan
- Center for Health Information and Communication, Department of Veterans Affairs, Health Services Research and Development Service, Veterans Health Administration, Richard L. Roudebush VA Medical Center, CIN 13 416, Indianapolis, IN, USA
| | - Katie Ernst
- Applied Decision Science LLC, Cincinnati, OH, USA
| | - Ann H Cottingham
- Indiana University Center for Health Services and Outcomes Research, Regenstrief Institute, Inc, 1101 West 10th Street, 46202-4800, Indianapolis, IN, USA
| | - Nicholas A Rattray
- Center for Health Information and Communication, Department of Veterans Affairs, Health Services Research and Development Service, Veterans Health Administration, Richard L. Roudebush VA Medical Center, CIN 13 416, Indianapolis, IN, USA
- Department of Medicine, Indiana University School of Medicine, Indianapolis, IN, USA
- Indiana University Center for Health Services and Outcomes Research, Regenstrief Institute, Inc, 1101 West 10th Street, 46202-4800, Indianapolis, IN, USA
| | - Zamal Franks
- Center for Health Information and Communication, Department of Veterans Affairs, Health Services Research and Development Service, Veterans Health Administration, Richard L. Roudebush VA Medical Center, CIN 13 416, Indianapolis, IN, USA
| | - April W Savoy
- Center for Health Information and Communication, Department of Veterans Affairs, Health Services Research and Development Service, Veterans Health Administration, Richard L. Roudebush VA Medical Center, CIN 13 416, Indianapolis, IN, USA
- Indiana University Center for Health Services and Outcomes Research, Regenstrief Institute, Inc, 1101 West 10th Street, 46202-4800, Indianapolis, IN, USA
- Purdue School of Engineering and Technology, Indiana University-Purdue University Indianapolis, Indianapolis, IN, USA
| | - Joy L Lee
- Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Worcester, MA, USA
| | - Richard M Frankel
- Center for Health Information and Communication, Department of Veterans Affairs, Health Services Research and Development Service, Veterans Health Administration, Richard L. Roudebush VA Medical Center, CIN 13 416, Indianapolis, IN, USA
- Department of Medicine, Indiana University School of Medicine, Indianapolis, IN, USA
- Indiana University Center for Health Services and Outcomes Research, Regenstrief Institute, Inc, 1101 West 10th Street, 46202-4800, Indianapolis, IN, USA
| |
Collapse
|
2
|
Harrington L. Comparison of Generative Artificial Intelligence and Predictive Artificial Intelligence. AACN Adv Crit Care 2024; 35:93-96. [PMID: 38848562 DOI: 10.4037/aacnacc2024225] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/09/2024]
Affiliation(s)
- Linda Harrington
- Linda Harrington is an Independent Consultant, Health Informatics and Digital Strategy, and Adjunct Professor at Texas Christian University, 2800 South University Drive, Fort Worth, TX 76109
| |
Collapse
|
3
|
Healy L, Mema B. Assessments to Ensure Quality of Notes during Transfer of Patient Care. ATS Sch 2024; 5:224-226. [PMID: 38957497 PMCID: PMC11216287 DOI: 10.34197/ats-scholar.2024-0045ed] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/04/2024] Open
Affiliation(s)
- Lydia Healy
- Department of Critical Care Medicine, Hospital for Sick Children, Toronto, Ontario, Canada; and
- Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Briseida Mema
- Department of Critical Care Medicine, Hospital for Sick Children, Toronto, Ontario, Canada; and
- Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
4
|
Lyons PG, Rojas JC, Bewley AF, Malone SM, Santhosh L. Validating the Physician Documentation Quality Instrument for Intensive Care Unit-Ward Transfer Notes. ATS Sch 2024; 5:274-285. [PMID: 39055332 PMCID: PMC11270237 DOI: 10.34197/ats-scholar.2023-0094oc] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Accepted: 01/09/2024] [Indexed: 07/27/2024] Open
Abstract
Background Physician communication failures during transfers of patients from the intensive care unit (ICU) to the general ward are common and can lead to adverse events. Efforts to improve written handoffs during these transfers are increasingly prominent, but no instruments have been developed to assess the quality of physician ICU-ward transfer notes. Objective To collect validity evidence for the modified nine-item Physician Documentation Quality Instrument (mPDQI-9) for assessing ICU-ward transfer note usefulness across several hospitals. Methods Twenty-four physician raters independently used the mPDQI-9 to grade 12 notes collected from three academic hospitals. A priori, we excluded the "up-to-date" and "accurate" domains, because these could not be assessed without giving the rater access to the complete patient chart. Assessments therefore used the domains "thorough," "useful," "organized," "comprehensible," "succinct," "synthesized," and "consistent." Raters scored each domain on a Likert scale ranging from 1 (low) to 5 (high). The total mPDQI-9 was the sum of these domain scores. The primary outcome was the raters' perceived clinical utility of the notes, and the primary measures of interest were criterion validity (Spearman's ρ) and interrater reliability (intraclass correlation [ICC]). Results Mean mPDQI-9 scores by note ranged from 19 (SD = 5.5) to 30 (SD = 4.2). Mean note ratings did not systematically differ by rater expertise (for interaction, P = 0.15). The proportion of raters perceiving each note as independently sufficient for patient care (the primary outcome) ranged from 33% to 100% across the set of notes. We found a moderately positive correlation between mPDQI-9 ratings and raters' overall assessments of each note's clinical utility (ρ = 0.48, P < 0.001). Interrater reliability was strong; the overall ICC was 0.89 (95% confidence interval [CI], 0.80-0.85), and ICCs were similar among reviewer groups. Finally, Cronbach's α was 0.87 (95% CI, 0.84-0.89), indicating good internal consistency. Conclusions We report moderate validity evidence for the mPDQI-9 to assess the usefulness of ICU-ward transfer notes written by internal medicine residents.
Collapse
Affiliation(s)
- Patrick G. Lyons
- Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, and
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon
| | - Juan C. Rojas
- Division of Pulmonary and Critical Care Medicine, Department of Internal Medicine, Rush University, Chicago, Illinois
| | - Alice F. Bewley
- Division of Infectious Diseases, Department of Medicine, and
| | - Sara M. Malone
- Division of Public Health Sciences, Department of Surgery, Washington University School of Medicine, St. Louis, Missouri; and
| | - Lekshmi Santhosh
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, University of California, San Francisco, San Francisco, California
| |
Collapse
|
5
|
Kyrios M, Levido J, Talbot D, Harris A. Off-label prescribing of psychotropics in a psychiatric patient population in Australia. Australas Psychiatry 2024; 32:196-200. [PMID: 38427939 DOI: 10.1177/10398562241237659] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/03/2024]
Abstract
BACKGROUND This study aimed to investigate the practice of off-label prescribing in both in- and outpatient psychiatry practice. METHODS One-hundred inpatient and 100 outpatient medical records from adult patients of an Australian psychiatry service from 2020 to 2021 were examined to determine the prevalence of off-label prescribing as defined by Therapeutic Goods Administration (TGA) indications, adherence to Royal Australian and New Zealand College of Psychiatrists (RANZCP) treatment guidelines, frequency of off-label prescription, and the quality of documentation and informed consent process. RESULTS Most prescribing events in both in- and outpatient settings were either on-label or off-label but consistent with RANZCP guidelines. Patients with a schizoaffective disorder diagnosis or displaying aggression were most likely to receive off-label prescriptions. There was no significant difference between in- and outpatient groups in the quality of documentation or consent process. CONCLUSIONS In general, off-label prescribing across groups was common, but many decisions were then in line with RANZCP recommendations. That there is a discrepancy between clinical and regulatory bodies has implications for how off-label status is decided.
Collapse
Affiliation(s)
- Mietta Kyrios
- Department of Psychiatry, Westmead Hospital, Wentworthville, NSW, Australia
| | - Jesse Levido
- Te Whatu Ora (Health New Zealand), Wellington, New Zealand
| | - Daniel Talbot
- Department of Psychiatry, Westmead Hospital, Wentworthville, NSW, Australia
| | - Anthony Harris
- Department of Psychiatry, Westmead Hospital, Wentworthville, NSW, Australia; Westmead Institute for Medical Research, Westmead, NSW, Australia; Westmead Clinical School, Sydney Medical School, University of Sydney, Westmead, NSW, Australia
| |
Collapse
|
6
|
Kernberg A, Gold JA, Mohan V. Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study. J Med Internet Res 2024; 26:e54419. [PMID: 38648636 DOI: 10.2196/54419] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Revised: 02/20/2024] [Accepted: 03/10/2024] [Indexed: 04/25/2024] Open
Abstract
BACKGROUND Medical documentation plays a crucial role in clinical practice, facilitating accurate patient management and communication among health care professionals. However, inaccuracies in medical notes can lead to miscommunication and diagnostic errors. Additionally, the demands of documentation contribute to physician burnout. Although intermediaries like medical scribes and speech recognition software have been used to ease this burden, they have limitations in terms of accuracy and addressing provider-specific metrics. The integration of ambient artificial intelligence (AI)-powered solutions offers a promising way to improve documentation while fitting seamlessly into existing workflows. OBJECTIVE This study aims to assess the accuracy and quality of Subjective, Objective, Assessment, and Plan (SOAP) notes generated by ChatGPT-4, an AI model, using established transcripts of History and Physical Examination as the gold standard. We seek to identify potential errors and evaluate the model's performance across different categories. METHODS We conducted simulated patient-provider encounters representing various ambulatory specialties and transcribed the audio files. Key reportable elements were identified, and ChatGPT-4 was used to generate SOAP notes based on these transcripts. Three versions of each note were created and compared to the gold standard via chart review; errors generated from the comparison were categorized as omissions, incorrect information, or additions. We compared the accuracy of data elements across versions, transcript length, and data categories. Additionally, we assessed note quality using the Physician Documentation Quality Instrument (PDQI) scoring system. RESULTS Although ChatGPT-4 consistently generated SOAP-style notes, there were, on average, 23.6 errors per clinical case, with errors of omission (86%) being the most common, followed by addition errors (10.5%) and inclusion of incorrect facts (3.2%). There was significant variance between replicates of the same case, with only 52.9% of data elements reported correctly across all 3 replicates. The accuracy of data elements varied across cases, with the highest accuracy observed in the "Objective" section. Consequently, the measure of note quality, assessed by PDQI, demonstrated intra- and intercase variance. Finally, the accuracy of ChatGPT-4 was inversely correlated to both the transcript length (P=.05) and the number of scorable data elements (P=.05). CONCLUSIONS Our study reveals substantial variability in errors, accuracy, and note quality generated by ChatGPT-4. Errors were not limited to specific sections, and the inconsistency in error types across replicates complicated predictability. Transcript length and data complexity were inversely correlated with note accuracy, raising concerns about the model's effectiveness in handling complex medical cases. The quality and reliability of clinical notes produced by ChatGPT-4 do not meet the standards required for clinical use. Although AI holds promise in health care, caution should be exercised before widespread adoption. Further research is needed to address accuracy, variability, and potential errors. ChatGPT-4, while valuable in various applications, should not be considered a safe alternative to human-generated clinical documentation at this time.
Collapse
Affiliation(s)
- Annessa Kernberg
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Sciences University, Portland, OR, United States
| | - Jeffrey A Gold
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Sciences University, Portland, OR, United States
| | - Vishnu Mohan
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Sciences University, Portland, OR, United States
| |
Collapse
|
7
|
Williams CY, Bains J, Tang T, Patel K, Lucas AN, Chen F, Miao BY, Butte AJ, Kornblith AE. Evaluating Large Language Models for Drafting Emergency Department Discharge Summaries. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.04.03.24305088. [PMID: 38633805 PMCID: PMC11023681 DOI: 10.1101/2024.04.03.24305088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/19/2024]
Abstract
Importance Large language models (LLMs) possess a range of capabilities which may be applied to the clinical domain, including text summarization. As ambient artificial intelligence scribes and other LLM-based tools begin to be deployed within healthcare settings, rigorous evaluations of the accuracy of these technologies are urgently needed. Objective To investigate the performance of GPT-4 and GPT-3.5-turbo in generating Emergency Department (ED) discharge summaries and evaluate the prevalence and type of errors across each section of the discharge summary. Design Cross-sectional study. Setting University of California, San Francisco ED. Participants We identified all adult ED visits from 2012 to 2023 with an ED clinician note. We randomly selected a sample of 100 ED visits for GPT-summarization. Exposure We investigate the potential of two state-of-the-art LLMs, GPT-4 and GPT-3.5-turbo, to summarize the full ED clinician note into a discharge summary. Main Outcomes and Measures GPT-3.5-turbo and GPT-4-generated discharge summaries were evaluated by two independent Emergency Medicine physician reviewers across three evaluation criteria: 1) Inaccuracy of GPT-summarized information; 2) Hallucination of information; 3) Omission of relevant clinical information. On identifying each error, reviewers were additionally asked to provide a brief explanation for their reasoning, which was manually classified into subgroups of errors. Results From 202,059 eligible ED visits, we randomly sampled 100 for GPT-generated summarization and then expert-driven evaluation. In total, 33% of summaries generated by GPT-4 and 10% of those generated by GPT-3.5-turbo were entirely error-free across all evaluated domains. Summaries generated by GPT-4 were mostly accurate, with inaccuracies found in only 10% of cases, however, 42% of the summaries exhibited hallucinations and 47% omitted clinically relevant information. Inaccuracies and hallucinations were most commonly found in the Plan sections of GPT-generated summaries, while clinical omissions were concentrated in text describing patients' Physical Examination findings or History of Presenting Complaint. Conclusions and Relevance In this cross-sectional study of 100 ED encounters, we found that LLMs could generate accurate discharge summaries, but were liable to hallucination and omission of clinically relevant information. A comprehensive understanding of the location and type of errors found in GPT-generated clinical text is important to facilitate clinician review of such content and prevent patient harm.
Collapse
Affiliation(s)
| | - Jaskaran Bains
- Department of Emergency Medicine; University of California, San Francisco
| | - Tianyu Tang
- Department of Emergency Medicine; University of California, San Francisco
| | - Kishan Patel
- Department of Emergency Medicine; University of California, San Francisco
| | - Alexa N. Lucas
- Department of Emergency Medicine; University of California, San Francisco
| | - Fiona Chen
- Department of Emergency Medicine; University of California, San Francisco
| | - Brenda Y. Miao
- Bakar Computational Health Sciences Institute; University of California, San Francisco
| | - Atul J. Butte
- Bakar Computational Health Sciences Institute; University of California, San Francisco
| | - Aaron E. Kornblith
- Bakar Computational Health Sciences Institute; University of California, San Francisco
- Department of Emergency Medicine; University of California, San Francisco
| |
Collapse
|
8
|
Ehmann MR, Murano T, Sullivan C, Egan DJ, Nazario S, Regan L. Remediation Methods 2.0 for Professionalism and Interpersonal and Communication Skills Milestones: An Update. J Grad Med Educ 2024; 16:128-132. [PMID: 38993300 PMCID: PMC11234317 DOI: 10.4300/jgme-d-23-00536.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 07/13/2024] Open
Affiliation(s)
- Michael R Ehmann
- is Assistant Professor and Residency Program Director, Department of Emergency Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Tiffany Murano
- is Professor and Vice Chair of Education, Department of Emergency Medicine, Columbia University Vagelos College of Physicians and Surgeons, New York, New York, USA
| | - Christine Sullivan
- is Professor and Associate Dean for Professional Development, Department of Emergency Medicine, University of Missouri-Kansas City School of Medicine, Kansas City, Missouri, USA
| | - Daniel J Egan
- is Program Director, Harvard Affiliated Emergency Medicine Residency, and Associate Professor and Vice Chair of Education, Massachusetts General Brigham Emergency Medicine, Boston, Massachusetts, USA
| | - Steven Nazario
- is Program Director, AdventHealth Emergency Medicine Residency, Orlando, Florida, USA; and
| | - Linda Regan
- is Associate Professor and Vice Chair for Education, Department of Emergency Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| |
Collapse
|
9
|
Baker HP, Dwyer E, Kalidoss S, Hynes K, Wolf J, Strelzow JA. ChatGPT's Ability to Assist with Clinical Documentation: A Randomized Controlled Trial. J Am Acad Orthop Surg 2024; 32:123-129. [PMID: 37976385 DOI: 10.5435/jaaos-d-23-00474] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Accepted: 09/29/2023] [Indexed: 11/19/2023] Open
Abstract
INTRODUCTION Clinical documentation is a critical aspect of health care that enables healthcare providers to communicate effectively with each other and maintain accurate patient care records. Artificial intelligence tools, such as chatbots and virtual assistants, have the potential to assist healthcare providers in clinical documentation. ChatGPT is an artificial intelligence conversational model that generates human-like responses to text-based prompts. In this study, we sought to investigate ChatGPT's ability to assist with writing a history of present illness based on standardized patient histories. METHODS A blinded, randomized controlled study was conducted to compare the use of typing, dictation, and ChatGPT as tools to document history of present illness (HPI) of standardized patient histories. Eleven study participants, consisting of medical students, orthopaedic surgery residents, and attending surgeons, completed three HPIs using a different documentation technique for each one. Participants were randomized into cohorts based on the type of documentation technique. Participants were asked to interview standardized patients and document the patient's history of present illness using their assigned method. RESULTS ChatGPT was found to be intermediate for speed; dictation was fastest, but produced markedly longer and higher quality patient histories based on Physician Documentation Quality Instrument score compared with dictation and typing. However, ChatGPT included erroneous information in 36% of the documents. Poor agreement existed on the quality of patient histories between reviewers. DISCUSSION Our study suggests that ChatGPT has the potential to improve clinical documentation by producing more comprehensive and organized HPIs. ChatGPT can generate longer and more detailed documentation compared with typing or dictation documentation methods. However, additional studies are needed to investigate and address concerns regarding privacy, bias, and accuracy of information.
Collapse
Affiliation(s)
- Hayden P Baker
- From the Department of Orthopaedic Surgery, University of Chicago, Chicago, IL
| | | | | | | | | | | |
Collapse
|
10
|
Ayer M. Relieving Administrative Burden on Clinical Staff with Streamlined Workflows And Speech-Recognition Software. BRITISH JOURNAL OF NURSING (MARK ALLEN PUBLISHING) 2023; 32:S1-S9. [PMID: 37708052 DOI: 10.12968/bjon.2023.32.sup16b.s4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/16/2023]
Affiliation(s)
- Mavis Ayer
- Multiple Sclerosis Lead Nurse, University Hospital Southampton NHS Foundation Trust
| |
Collapse
|
11
|
Friedes BD, McRae AE, Abdul-Raheem J, Balighian E, Golden W, Pahwa AK. Medical Student Note Quality on a Pediatrics Core Clerkship Differs by Service. Cureus 2023; 15:e44740. [PMID: 37809116 PMCID: PMC10557370 DOI: 10.7759/cureus.44740] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/03/2023] [Indexed: 10/10/2023] Open
Abstract
Introduction Medical students rotate on various clinical disciplines with the same professional goal of learning medical documentation. This study investigated differences between medical student notes on inpatient general and subspecialty pediatric services by comparing note quality, length, and file time. Methods In a single-site, observational cohort study, medical students in the Core Clerkship in Pediatrics (CCP) from July 2020 to June 2021 participated in a note-writing didactic course. We compared notes from medical students completing their inpatient assignment on a general pediatric service to those who completed it on a pediatric subspecialty service. Primary outcomes were note quality measured by Physician Documentation Quality Instrument-9 (PDQI9), note length (measured by line count), and file time (measured by hours to completion since 6 AM on the morning of note initiation). Results We evaluated 84 notes from 84 medical students on the general pediatric services and 50 notes from 49 medical students on the pediatric subspecialty services. Note quality measured by PDQI9 was significantly higher for general pediatric service notes compared to pediatric subspecialty service notes (p = 0.03). General pediatric service notes were significantly shorter (p < 0.001). We found no difference in file time (p = 0.23). Conclusion Medical student notes on pediatric subspecialty services scored significantly lower in quality and were longer compared to general pediatric services, demonstrating the need for a more tailored note-writing curriculum and note template based on service.
Collapse
Affiliation(s)
| | - Ashlyn E McRae
- Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, USA
| | | | | | | | - Amit K Pahwa
- Internal Medicine - Pediatrics, Johns Hopkins University, Baltimore, USA
| |
Collapse
|
12
|
McRae AE, Rowe JT, Friedes BD, Abdul-Raheem J, Balighian ED, Bertram A, Huang V, McFarland SR, McDaniel LM, Kumra T, Christopher Golden W, Pahwa AK. Assessing the Impact of a Note-Writing Session and Standardized Note Template on Medical Student Note Length and Quality. Acad Pediatr 2023; 23:1454-1458. [PMID: 36907434 DOI: 10.1016/j.acap.2023.02.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/19/2022] [Revised: 02/09/2023] [Accepted: 02/27/2023] [Indexed: 03/14/2023]
Abstract
PURPOSE To determine whether a formal note-writing session and note template for medical students (MS) during the Core Clerkship in Pediatrics (CCP) increase note quality, shortens note length, and decreases time of documentation. METHODS In this single site, prospective study, MS participating in an 8-week CCP received a didactic session on note-writing in the electronic health record (EHR) and utilized EHR template developed for the study. We assessed note quality (measured by Physician Documentation Quality Instrument-9 [PDQI-9]), note length and note documentation time in this group compared to MS notes on the CCP in the prior academic year. We used descriptive statistics and Kruskal-Wallis tests for analysis. RESULTS We analyzed 121 notes written by 40 students in the control group and 92 notes writing by 41 students in the intervention group. Notes from the intervention group were more "up to date," "accurate," "organized," and "comprehensible" compared to the control group (P = 0.02, P = 0.04, P = 0.01, and P = 0.02, respectively). Intervention group notes received higher cumulative PDQI-9 scores compared to the control group (median score 38 (IQR 34-42) versus 36 (IQR 32-40) out of 45 total, P = 0.04). Intervention group notes were approximately 35% shorter than the control group notes (median 68.5 lines vs 105 lines, P < 0.0001) and were signed earlier than control group notes (median file time 316 minute vs 352 minute, P = 0.02). CONCLUSIONS The intervention successfully decreased note length, improved note quality based on standardized metrics, and reduced time to completion of note documentation.
Collapse
Affiliation(s)
- Ashlyn E McRae
- Johns Hopkins University School of Medicine (AE McRae, JT Rowe, BD Friedes, J Abdul-Raheem, and A Bertram), Baltimore, Md.
| | - Julian T Rowe
- Johns Hopkins University School of Medicine (AE McRae, JT Rowe, BD Friedes, J Abdul-Raheem, and A Bertram), Baltimore, Md
| | - Barbara D Friedes
- Johns Hopkins University School of Medicine (AE McRae, JT Rowe, BD Friedes, J Abdul-Raheem, and A Bertram), Baltimore, Md
| | - Jareatha Abdul-Raheem
- Johns Hopkins University School of Medicine (AE McRae, JT Rowe, BD Friedes, J Abdul-Raheem, and A Bertram), Baltimore, Md
| | - Eric D Balighian
- Department of Pediatrics (ED Balighian, SR McFarland, T Kumra, and W. Christopher Golden), Johns Hopkins University School of Medicine, Baltimore, Md
| | - Amanda Bertram
- Johns Hopkins University School of Medicine (AE McRae, JT Rowe, BD Friedes, J Abdul-Raheem, and A Bertram), Baltimore, Md
| | - Victoria Huang
- Department of Anesthesia and Critical Care Medicine (V Huang), Johns Hopkins University School of Medicine, Baltimore, Md
| | - Susan R McFarland
- Department of Pediatrics (ED Balighian, SR McFarland, T Kumra, and W. Christopher Golden), Johns Hopkins University School of Medicine, Baltimore, Md
| | - Lauren M McDaniel
- Department of Pediatrics (LM McDaniel), University of Washington School of Medicine, Seattle, Wash
| | - Tina Kumra
- Department of Pediatrics (ED Balighian, SR McFarland, T Kumra, and W. Christopher Golden), Johns Hopkins University School of Medicine, Baltimore, Md
| | - William Christopher Golden
- Department of Pediatrics (ED Balighian, SR McFarland, T Kumra, and W. Christopher Golden), Johns Hopkins University School of Medicine, Baltimore, Md
| | - Amit K Pahwa
- Departments of Internal Medicine and Pediatrics (A Pahwa), The Johns Hopkins University School of Medicine, Baltimore, Md
| |
Collapse
|
13
|
Maria M, Maram K, Sarib H, Jason R, Eguale T, Mark L, Gordon SD. Assessing the Assessment-Developing and Deploying a Novel Tool for Evaluating Clinical Notes' Diagnostic Assessment Quality. J Gen Intern Med 2023; 38:2123-2129. [PMID: 36854867 PMCID: PMC10361936 DOI: 10.1007/s11606-023-08085-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 02/01/2023] [Indexed: 03/02/2023]
Abstract
BACKGROUND Ambulatory diagnostic errors are increasingly being recognized as an important quality and safety issue, and while measures of diagnostic quality have been sought, tools to evaluate diagnostic assessments in the medical record are lacking. OBJECTIVE To develop and test a tool to measure diagnostic assessment note quality in primary care urgent encounters and identify common elements and areas for improvement in diagnostic assessment. DESIGN Retrospective chart review of urgent care encounters at an urban academic setting. PARTICIPANTS Primary care physicians. MAIN MEASURES The Assessing the Assessment (ATA) instrument was evaluated for inter-rater reliability, internal consistency, and findings from its application to EHR notes. KEY RESULTS ATA had reasonable performance characteristics (kappa 0.63, overall Cronbach's alpha 0.76). Variability in diagnostic assessment was seen in several domains. Two components of situational awareness tended to be well-documented ("Don't miss diagnoses" present in 84% of charts, red flag symptoms in 87%), while Psychosocial context was present only 18% of the time. CONCLUSIONS The ATA tool is a promising framework for assessing and identifying areas for improvement in diagnostic assessments documented in clinical encounters.
Collapse
Affiliation(s)
- Mirica Maria
- Center for Patient Safety Research and Practice, Department of Medicine, Brigham and Women's Hospital, 3Rd Floor General Medicine, 1620 Tremont St, Boston, MA, 02120, USA
| | | | | | | | - Tewodros Eguale
- Center for Patient Safety Research and Practice, Department of Medicine, Brigham and Women's Hospital, 3Rd Floor General Medicine, 1620 Tremont St, Boston, MA, 02120, USA
- Massachusetts College of Pharmacy and Health Sciences (MCPHS), Boston, MA, USA
| | | | - Schiff D Gordon
- Center for Patient Safety Research and Practice, Department of Medicine, Brigham and Women's Hospital, 3Rd Floor General Medicine, 1620 Tremont St, Boston, MA, 02120, USA.
- Harvard Medical School, Center for Primary Care, Boston, MA, USA.
| |
Collapse
|
14
|
Wei M, Salgado E, Girard CE, Santoro JD, Lepore N. Your note, your way: how to write an inpatient progress note accurately and efficiently as an intern. Postgrad Med J 2023; 99:492-497. [PMID: 37294720 DOI: 10.1136/postgradmedj-2022-141834] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Accepted: 05/30/2022] [Indexed: 11/04/2022]
Abstract
A physician's progress note is an essential piece of documentation regarding key events and the daily status of patients during their hospital stay. It serves not only as a communication tool between care team members, but also chronicles clinical status and pertinent updates to their medical care. Despite the importance of these documents, little literature exists on how to help residents to improve the quality of their daily progress notes. A narrative literature review of English language literature was performed and summated to provide recommendations on how to write an inpatient progress note more accurately and efficiently. In addition, the authors will also introduce a method to build a personal template with the goal of extracting relevant data automatically to reduce clicks for an inpatient progress note in the electronic medical record system.
Collapse
Affiliation(s)
- Miao Wei
- Internal Medicine, Cleveland Clinic Florida, Weston, Florida, USA
| | - Efrain Salgado
- Department of Neurology, Cleveland Clinic Florida, Weston, Florida, USA
| | - Christine E Girard
- Department of Internal Medicine, Cleveland Clinic Florida, Weston, Florida, USA
| | - Jonathan D Santoro
- Department of Neurology, University of Southern California, Los Angeles, California, USA
| | - Natasha Lepore
- Department of Radiology, University of Southern California, Los Angeles, California, USA
| |
Collapse
|
15
|
Khazen M, Sullivan EE, Arabadjis S, Ramos J, Mirica M, Olson A, Linzer M, Schiff GD. How does work environment relate to diagnostic quality? A prospective, mixed methods study in primary care. BMJ Open 2023; 13:e071241. [PMID: 37147090 PMCID: PMC10163453 DOI: 10.1136/bmjopen-2022-071241] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/07/2023] Open
Abstract
OBJECTIVES The quest to measure and improve diagnosis has proven challenging; new approaches are needed to better understand and measure key elements of the diagnostic process in clinical encounters. The aim of this study was to develop a tool assessing key elements of the diagnostic assessment process and apply it to a series of diagnostic encounters examining clinical notes and encounters' recorded transcripts. Additionally, we aimed to correlate and contextualise these findings with measures of encounter time and physician burnout. DESIGN We audio-recorded encounters, reviewed their transcripts and associated them with their clinical notes and findings were correlated with concurrent Mini Z Worklife measures and physician burnout. SETTING Three primary urgent-care settings. PARTICIPANTS We conducted in-depth evaluations of 28 clinical encounters delivered by seven physicians. RESULTS Comparing encounter transcripts with clinical notes, in 24 of 28 (86%) there was high note/transcript concordance for the diagnostic elements on our tool. Reliably included elements were red flags (92% of notes/encounters), aetiologies (88%), likelihood/uncertainties (71%) and follow-up contingencies (71%), whereas psychosocial/contextual information (35%) and mentioning common pitfalls (7%) were often missing. In 22% of encounters, follow-up contingencies were in the note, but absent from the recorded encounter. There was a trend for higher burnout scores being associated with physicians less likely to address key diagnosis items, such as psychosocial history/context. CONCLUSIONS A new tool shows promise as a means of assessing key elements of diagnostic quality in clinical encounters. Work conditions and physician reactions appear to correlate with diagnostic behaviours. Future research should continue to assess relationships between time pressure and diagnostic quality.
Collapse
Affiliation(s)
- Maram Khazen
- Harvard Medical School, Center for Primary Care, Boston, Massachusetts, USA
- The Max Stern Yezreel Valley College, Emek Yezreel, Northern, Israel
| | - Erin E Sullivan
- Suffolk University Sawyer Business School, Boston, Massachusetts, USA
- Harvard Medical School Department of Global Health and Social Medicine, Boston, Massachusetts, USA
| | - Sophia Arabadjis
- University of California Santa Barbara, Santa Barbara, California, USA
| | - Jason Ramos
- Emory University School of Medicine, Atlanta, Georgia, USA
| | - Maria Mirica
- Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Andrew Olson
- University of Minnesota Medical School Twin Cities, Minneapolis, Minnesota, USA
| | - Mark Linzer
- Hennepin Healthcare System Inc, Minneapolis, Minnesota, USA
| | - Gordon D Schiff
- Harvard Medical School, Center for Primary Care, Boston, Massachusetts, USA
- Brigham and Women's Hospital, Boston, Massachusetts, USA
| |
Collapse
|
16
|
Feldman J, Goodman A, Hochman K, Chakravartty E, Austrian J, Iturrate E, Bosworth B, Saxena A, Moussa MM, Chenouda DM, Volpicelli F, Adler N, Weisstuch J, Testa P. Novel Note Templates to Enhance Signal and Reduce Noise in Medical Documentation: a Prospective Improvement Study. JMIR Form Res 2023; 7:e41223. [PMID: 36821760 PMCID: PMC10134024 DOI: 10.2196/41223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 01/23/2023] [Accepted: 02/15/2023] [Indexed: 02/17/2023] Open
Abstract
BACKGROUND The introduction of electronic workflows has allowed for the flow of raw un-contextualized clinical data into medical documentation. As a result, many electronic notes have become replete of "noise" and deplete of clinically significant "signals". There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating. OBJECTIVE To describe the development and impact of a novel set of templates designed to change the flow of information in medical documentation. METHODS This is a multi-hospital nonrandomized prospective improvement study conducted on the Inpatient General Internal Medicine Service across three hospital campuses at the New York University (NYU) Langone Health System. A group of physician leaders representing each campus met biweekly for six months. The output of these meetings included 1) a conceptualization of the note bloat problem as a dysfunction in information flow 2) a set of guiding principles for organizational documentation improvement 3) the design and build of novel electronic templates that reduced the flow of extraneous information into provider notes by providing link outs to best practice data visualizations and 4) a documentation improvement curriculum for inpatient medicine providers. Prior to go-live, pragmatic usability testing was performed with the new progress note template, and the overall user experience measured using the System Usability Scale (SUS). Primary outcomes measures after go-live include template utilization rate and note length in characters. RESULTS In usability testing amongst 22 medicine providers, the new progress note template averaged a usability score of 90.6/100 on the System Usability Scale. 77% of providers strongly agreed that the new template was easy to use. 68% strongly agreed that they would like to use the template frequently. In the three months after template implementation, General Internal Medicine providers wrote 65% of all inpatient notes with the new templates. During this period of time the organization saw a 46%, 47%, and 32% reduction in note length for general medicine progress notes, consults, and H&Ps, respectively, when compared to a baseline measurement period prior to interventions. CONCLUSIONS A bundled intervention that included deployment of novel templates for inpatient general medicine providers significantly reduced average note length on the clinical service. Templates designed to reduce the flow of extraneous information into provider notes performed well during usability testing, and these templates were rapidly adopted across all hospital campuses. Further research is needed to assess the impact of novel templates on note quality, provider efficiency and patient outcomes. CLINICALTRIAL
Collapse
Affiliation(s)
- Jonah Feldman
- Medical Center Information Technology, NYU Langone Health, New York, US.,Department of Medicine, NYU Long Island School of Medicine, Mineola, US
| | - Adam Goodman
- Division of Gastroenterology & Hepatology, NYU Grossman School of Medicine, New York,, US
| | - Katherine Hochman
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Eesha Chakravartty
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US.,Medical Center Information Technology, NYU Langone Health, New York, US
| | - Jonathan Austrian
- Medical Center Information Technology, NYU Langone Health, New York, US.,Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Eduardo Iturrate
- Medical Center Information Technology, NYU Langone Health, New York, US.,Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Brian Bosworth
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Archana Saxena
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Marwa M Moussa
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | - Dina M Chenouda
- Department of Medicine, NYU Long Island School of Medicine, Mineola, US
| | | | - Nicole Adler
- Department of Medicine, New York University Langone Health, 550 1st avenue, New York, US
| | | | - Paul Testa
- Medical Center Information Technology, NYU Langone Health, New York, US
| |
Collapse
|
17
|
Eshel R, Bellolio F, Boggust A, Shapiro NI, Mullan AF, Heaton HA, Madsen BE, Homme JL, Iliff BW, Sunga KL, Wangsgard CR, Vanmeter D, Cabrera D. Comparison of clinical note quality between an automated digital intake tool and the standard note in the emergency department. Am J Emerg Med 2023; 63:79-85. [PMID: 36327754 DOI: 10.1016/j.ajem.2022.10.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2022] [Revised: 09/05/2022] [Accepted: 10/07/2022] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Medical encounters require an efficient and focused history of present illness (HPI) to create differential diagnoses and guide diagnostic testing and treatment. Our aim was to compare the HPI of notes created by an automated digital intake tool versus standard medical notes created by clinicians. METHODS Prospective trial in a quaternary academic Emergency Department (ED). Notes were compared using the 5-point Physician Documentation Quality Instrument (PDQI-9) scale and the Centers for Medicare & Medicaid Services (CMS) level of complexity index. Reviewers were board certified emergency medicine physicians blinded to note origin. Reviewers received training and calibration prior to note assessments. A difference of 1 point was considered clinically significant. Analysis included McNemar's (binary), Wilcoxon-rank (Likert), and agreement with Cohen's Kappa. RESULTS A total of 148 ED medical encounters were charted by both digital note and standard clinical note. The ability to capture patient information was assessed through comparison of note content across paired charts (digital-standard note on the same patient), as well as scores given by the reviewers. Reviewer agreement was kappa 0.56 (CI 0.49-0.64), indicating moderate level of agreement between reviewers scoring the same patient chart. Considering all 18 questions across PDQI-9 and CMS scales, the average agreement between standard clinical note and digital note was 54.3% (IQR 44.4-66.7%). There was a moderate level of agreement between content of standard and digital notes (kappa 0.54, 95%CI 0.49-0.60). The quality of the digital note was within the 1 point clinically significant difference for all of the attributes, except for conciseness. Digital notes had a higher frequency of CMS severity elements identified. CONCLUSION Digitally generated clinical notes had moderate agreement compared to standard clinical notes and within the one point clinically significant difference except for the conciseness attribute. Digital notes more reliably documented billing components of severity. The use of automated notes should be further explored to evaluate its utility in facilitating documentation of patient encounters.
Collapse
Affiliation(s)
- Ron Eshel
- Department of Anesthesia, Critical Care and Pain, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel
| | - Fernanda Bellolio
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
| | - Andy Boggust
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
| | - Nathan I Shapiro
- Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, MA, United States; Diagnostics Robotics. Tel Aviv, Israel
| | - Aidan F Mullan
- Department of Health Sciences Research, Division of Health Care Policy and Research, Mayo Clinic, Rochester, MN, United States
| | - Heather A Heaton
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
| | - Bo E Madsen
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
| | - James L Homme
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
| | - Benjamin W Iliff
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
| | - Kharmene L Sunga
- Department of Anesthesia, Critical Care and Pain, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel
| | | | - Derek Vanmeter
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States
| | - Daniel Cabrera
- Department of Emergency Medicine, Mayo Clinic, Rochester, MN, United States.
| |
Collapse
|
18
|
Eysenbach G, Wark PA, Mastellos N, Neves AL, Gallagher J, Majeed A, Webster A, Smith A, Choo-Kang B, Leon C, Edwards C, O'Shea C, Heitz E, Kayode OV, Nash M, Kowalski M, Jiwani M, O'Callaghan ME, Zary N, Henderson N, Chavannes NH, Čivljak R, Olubiyi OA, Mahapatra P, Panday RN, Oriji SO, Fox TE, Faint V, Car J. Assessment of Clinical Information Quality in Digital Health Technologies: International eDelphi Study. J Med Internet Res 2022; 24:e41889. [PMID: 36472901 PMCID: PMC9768639 DOI: 10.2196/41889] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2022] [Revised: 10/29/2022] [Accepted: 11/17/2022] [Indexed: 11/18/2022] Open
Abstract
BACKGROUND Digital health technologies (DHTs), such as electronic health records and prescribing systems, are transforming health care delivery around the world. The quality of information in DHTs is key to the quality and safety of care. We developed a novel clinical information quality (CLIQ) framework to assess the quality of clinical information in DHTs. OBJECTIVE This study explored clinicians' perspectives on the relevance, definition, and assessment of information quality dimensions in the CLIQ framework. METHODS We used a systematic and iterative eDelphi approach to engage clinicians who had information governance roles or personal interest in information governance; the clinicians were recruited through purposive and snowball sampling techniques. Data were collected using semistructured online questionnaires until consensus was reached on the information quality dimensions in the CLIQ framework. Responses on the relevance of the dimensions were summarized to inform decisions on retention of the dimensions according to prespecified rules. Thematic analysis of the free-text responses was used to revise definitions and the assessment of dimensions. RESULTS Thirty-five clinicians from 10 countries participated in the study, which was concluded after the second round. Consensus was reached on all dimensions and categories in the CLIQ framework: informativeness (accuracy, completeness, interpretability, plausibility, provenance, and relevance), availability (accessibility, portability, security, and timeliness), and usability (conformance, consistency, and maintainability). A new dimension, searchability, was introduced in the availability category to account for the ease of finding needed information in the DHTs. Certain dimensions were renamed, and some definitions were rephrased to improve clarity. CONCLUSIONS The CLIQ framework reached a high expert consensus and clarity of language relating to the information quality dimensions. The framework can be used by health care managers and institutions as a pragmatic tool for identifying and forestalling information quality problems that could compromise patient safety and quality of care. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) RR2-10.1136/bmjopen-2021-057430.
Collapse
Affiliation(s)
| | - Petra A Wark
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom.,Centre for Intelligent Healthcare, Institute for Health and Wellbeing, Coventry University, Coventry, United Kingdom
| | - Nikolaos Mastellos
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
| | - Ana Luisa Neves
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
| | - Joseph Gallagher
- gHealth Research Group, School of Medicine, University College Dublin, Dublin, Ireland
| | - Azeem Majeed
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
| | - Andrew Webster
- Leeds Teaching Hospital National Health Service Trust, Leeds, United Kingdom
| | - Anthony Smith
- Health Board, Hywel Dda University, Wales, United Kingdom
| | - Brian Choo-Kang
- Glasgow Royal Infirmary, National Health Service Greater Glasgow and Clyde, Glasgow, United Kingdom
| | - Catherine Leon
- Oxford University National Health Service Foundation Trusts, Oxford, United Kingdom
| | | | - Conor O'Shea
- Wheaton Hall Medical Practice, Drogheda, Co Louth, Ireland
| | | | | | - Makeba Nash
- Croydon Health Services, Croydon, United Kingdom
| | - Martin Kowalski
- Department of Ophthalmology, University Eye Hospital Tübingen, Tübingen, Germany
| | - Mateen Jiwani
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
| | | | - Nabil Zary
- Institute for Excellence in Health Professions Education, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates
| | | | - Niels H Chavannes
- National eHealth Living Lab, Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, Netherlands
| | - Rok Čivljak
- University Hospital for Infectious Diseases, Zagreb, Croatia.,University of Zagreb School of Medicine, Zagreb, Croatia
| | - Olubunmi Abiola Olubiyi
- Medical Research Council Unit The Gambia, London School of Hygiene & Tropical Medicine, Fajara, Gambia
| | - Piyush Mahapatra
- West Hertfordshire Teaching Hospitals National Health Service Trust, Watford, United Kingdom
| | - Rishi Nannan Panday
- Acute Medicine Section, Department of Internal Medicine, Amsterdam University Medical Centers, VU Medical Center Location, Amsterdam, Netherlands
| | - Sunday O Oriji
- Department of Mental Health, Nnewi Campus, Nnamdi Azikiwe University Teaching Hospital, Nnewi, Nigeria
| | - Tatiana Erlikh Fox
- Centre for Population Health Sciences, Lee Kong Chian Medicine, Nanyang Technological University, Singapore, Singapore
| | - Victoria Faint
- Leeds Teaching Hospital National Health Service Trust, Leeds, United Kingdom
| | - Josip Car
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom.,Centre for Population Health Sciences, Lee Kong Chian Medicine, Nanyang Technological University, Singapore, Singapore
| |
Collapse
|
19
|
Iscoe MS, McLean RM, Melnick ER. Restoring Meaningful Content to the Medical Record: Standardizing Measurement Could Improve EHR Utility While Decreasing Burden. Mayo Clin Proc 2022; 97:1971-1974. [PMID: 36210197 DOI: 10.1016/j.mayocp.2022.07.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Revised: 07/02/2022] [Accepted: 07/14/2022] [Indexed: 11/05/2022]
Affiliation(s)
- Mark S Iscoe
- Department of Emergency Medicine, Yale School of Medicine, New Haven, CT.
| | - Robert M McLean
- Department of Internal Medicine, Yale School of Medicine, New Haven, CT; Northeast Medical Group, Stratford, CT
| | - Edward R Melnick
- Department of Emergency Medicine, Yale School of Medicine, New Haven, CT; Department of Biostatistics (Health Informatics), Yale School of Public Health, New Haven, CT
| |
Collapse
|
20
|
Richardson KM, Cristiano JA, Schafer KR, Shen E, Burns CA. Writing Is Thinking: Implementation and Evaluation of an Internal Medicine Residency Clinical Reasoning and Documentation Curriculum. MEDICAL SCIENCE EDUCATOR 2022; 32:773-777. [PMID: 36035531 PMCID: PMC9411408 DOI: 10.1007/s40670-022-01570-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/29/2022] [Indexed: 06/15/2023]
Abstract
UNLABELLED With increasingly complicated patients and faster throughput, time for thorough critical thinking and thoughtful clinical documentation is limited, especially in the training environment. Advocating for the value of clinical documentation as a robust opportunity for critical thinking, we describe the implementation and evaluation of a clinical reasoning and documentation curriculum for internal medicine residents. Our curriculum employed facilitated discussion, practical application, and a resident-as-teacher model. Resident surveys showed improved perceptions of the clinical and educational value of clinical documentation. Residents reported increased feedback to interns about their documentation and more appreciation of documentation as a venue for critical thinking. SUPPLEMENTARY INFORMATION The online version contains supplementary material available at 10.1007/s40670-022-01570-5.
Collapse
Affiliation(s)
- Karl M. Richardson
- Wake Forest School of Medicine, Medical Center Dr, Winston Salem, NC 27157 USA
| | - Joseph A. Cristiano
- Wake Forest School of Medicine, Medical Center Dr, Winston Salem, NC 27157 USA
| | | | - E. Shen
- Wake Forest School of Medicine, Medical Center Dr, Winston Salem, NC 27157 USA
| | - Cynthia A. Burns
- Wake Forest School of Medicine, Medical Center Dr, Winston Salem, NC 27157 USA
| |
Collapse
|
21
|
Schaye V, Guzman B, Burk-Rafel J, Marin M, Reinstein I, Kudlowitz D, Miller L, Chun J, Aphinyanaphongs Y. Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning Documentation. J Gen Intern Med 2022; 37:2230-2238. [PMID: 35710676 PMCID: PMC9296753 DOI: 10.1007/s11606-022-07526-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 03/29/2022] [Indexed: 11/29/2022]
Abstract
BACKGROUND Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment. OBJECTIVE The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes. DESIGN, PARTICIPANTS, MAIN MEASURES Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend. KEY RESULTS The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications). CONCLUSIONS The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.
Collapse
Affiliation(s)
- Verity Schaye
- NYU Grossman School of Medicine, New York, NY, USA. .,NYC Health & Hospitals/Bellevue, New York, NY, USA.
| | | | | | - Marina Marin
- NYU Grossman School of Medicine, New York, NY, USA
| | | | | | - Louis Miller
- Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| | - Jonathan Chun
- Stanford University School of Medicine, Stanford, CA, USA
| | | |
Collapse
|
22
|
Kim M, Chan N, Evans J, Min JK, Hayton AC. Improving Medical Student Inpatient Documentation Through Feedback Using a Note Assessment Tool. Cureus 2022; 14:e23369. [PMID: 35475068 PMCID: PMC9020806 DOI: 10.7759/cureus.23369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/11/2022] [Indexed: 11/05/2022] Open
Abstract
Introduction Documentation within the Electronic Health Record (EHR) is an essential skill for medical students to succeed in residency and post-residency training. The increased use of medical student progress notes for billable services raises the need for the education and assessment of quality note writing. We hypothesized that structured note feedback using a note assessment tool would improve the quality of medical student inpatient progress notes. Methods We conducted a retrospective study to review the quality of student inpatient progress notes written before and after structured feedback using the Responsible Electronic Documentation (RED) checklist throughout a third-year internal medicine clerkship. The first intervention group received feedback from clerkship directors in the 2017-2018 academic year and the second intervention group received feedback from ward residents/attendings in the 2018-2019 academic year. Within each intervention group, the total note scores from pre and post-intervention were compared. Results Feedback from clerkship directors yielded a greater increase in students’ total note score from pre to post-intervention compared to ward resident/attending feedback (F(1,255) = 12.84, p < 0.001). Cohen’s d effect size value was greater for the clerkship director feedback arm (d=0.71) compared to the ward resident/attending feedback arm (d=0.24). Post-hoc analyses using dependent sample t-tests revealed that there were significant increases in total note scores from pre to post-intervention for both the clerkship director arm (t(123) = 8.26, p < 0.001, d = 0.71) and the ward resident/attending arm (t(132) = 2.85, p = 0.005, d = 0.24). Conclusion Clerkship director feedback led to a greater increase in medical student documentation compared to ward attending/resident feedback. Nonetheless, structured feedback with a note assessment tool, whether from clerkship directors or ward attendings/residents, leads to a significant improvement in medical student documentation. Though there are various methods for providing feedback, educators can use the RED checklist to provide clear guidelines that will facilitate note-writing feedback.
Collapse
|
23
|
Vawdrey DK, Cauthorn C, Francis D, Hackenberg K, Maloney G, Hohmuth BA. A Practical Approach for Monitoring the Use of Copy-Paste in Clinical Notes. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2022; 2021:1178-1185. [PMID: 35308931 PMCID: PMC8861699] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
The use of copy-paste in authoring clinical notes has been widely embraced by busy providers, but inappropriate copy-paste has been lambasted by critics for introducing risks related to patient safety and regulatory compliance. At an integrated academic health system with over 4,100 providers writing notes, we developed a pragmatic approach to assess the use of copy-paste. From January 1-December 31, 2020, approximately 2.3M inpatient notes and 6.6M ambulatory clinic notes were authored in our electronic health record. Of the inpatient notes, 42% used copy-paste, and 19% of overall note content was copied; in ambulatory notes, 18% used copy-paste and 12% of note content was copied. We describe an approach for including providers' copy-paste usage statistics into the ongoing professional practice evaluation process required for hospital accreditation, thereby offering individual training opportunities related to the lack of use of copy-paste or its potential overuse.
Collapse
Affiliation(s)
- David K Vawdrey
- Geisinger Steele Institute for Health Innovation, Danville, PA
- Columbia University Department of Biomedical Informatics, New York, NY
| | - Casey Cauthorn
- Geisinger Steele Institute for Health Innovation, Danville, PA
| | - Diane Francis
- Geisinger Steele Institute for Health Innovation, Danville, PA
| | | | | | - Benjamin A Hohmuth
- Geisinger Steele Institute for Health Innovation, Danville, PA
- Geisinger Department of Medicine, Danville, PA
| |
Collapse
|
24
|
Schaye V, Miller L, Kudlowitz D, Chun J, Burk-Rafel J, Cocks P, Guzman B, Aphinyanaphongs Y, Marin M. Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback. J Gen Intern Med 2022; 37:507-512. [PMID: 33945113 PMCID: PMC8858363 DOI: 10.1007/s11606-021-06805-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/10/2020] [Accepted: 04/03/2021] [Indexed: 02/03/2023]
Abstract
BACKGROUND Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability. OBJECTIVE Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool. DESIGN, PARTICIPANTS, AND MAIN MEASURES The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting. KEY RESULTS The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality. CONCLUSIONS The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.
Collapse
Affiliation(s)
- Verity Schaye
- NYU Grossman School of Medicine, New York, NY, USA. .,NYC Health + Hospitals/Bellevue, New York, NY, USA.
| | - Louis Miller
- Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| | | | - Jonathan Chun
- Stanford University School of Medicine, Stanford, CA, USA
| | | | | | | | | | - Marina Marin
- NYU Grossman School of Medicine, New York, NY, USA
| |
Collapse
|
25
|
Cristiano JA, Jackson JM, Shen E, Williams DM, Ellis LR. Integrating the Electronic Health Record Into Patient Encounters: An Introductory Standardized Patient Exercise for Preclinical Medical Students. MEDEDPORTAL : THE JOURNAL OF TEACHING AND LEARNING RESOURCES 2022; 18:11209. [PMID: 35047666 PMCID: PMC8727442 DOI: 10.15766/mep_2374-8265.11209] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/24/2021] [Accepted: 09/25/2021] [Indexed: 05/12/2023]
Abstract
INTRODUCTION Increasingly, use of the electronic health record (EHR) is interwoven into even the most basic patient care tasks. Accordingly, learning how to utilize the EHR during patient encounters is important for medical students as they develop their clinical skills. Existing EHR curricula have focused primarily on doctor-patient relationship skills. We developed a session for our preclinical students on EHR-related doctor-patient relationship skills as well as on using the EHR to verify data and focus one's history taking. METHODS We developed student notes, three training videos, four standardized patient (SP) cases, and a simplified, simulated EHR based on these cases. Students reviewed the notes and videos prior to class. During class, students practiced EHR-related communication and data-collection strategies by interviewing an SP while interacting with the simulated EHR. Following each encounter, students received feedback from a small group of peers and faculty. RESULTS Two-hundred eighty-nine second-year medical students participated this session in 2019 and 2020, and 27 (19%, 2019) and 40 (28%, 2020) students, respectively, completed the postsession evaluation. Most respondents rated the SP activity as extremely or quite effective for practicing doctor-patient relationship strategies while interacting with the EHR (89%, 2019; 83%, 2020) and for practicing verification of EHR data during a patient encounter (81%, 2019; 86%, 2020). DISCUSSION This training session was effective for introducing preclinical medical students to fundamental concepts and skills related to incorporating the EHR into patient encounters and offers a low-cost approach to teaching early medical students these important skills.
Collapse
Affiliation(s)
- Joseph A. Cristiano
- Assistant Professor, Department of Internal Medicine, Section of General Internal Medicine, Wake Forest School of Medicine
- Corresponding author:
| | - Jennifer M. Jackson
- Associate Professor, Department of Pediatrics, Section of Pediatric Hospital Medicine, Wake Forest School of Medicine
| | - E Shen
- Assistant Professor, Department of Internal Medicine, Section of General Internal Medicine, Wake Forest School of Medicine
| | - Donna M. Williams
- Associate Professor, Department of Internal Medicine, Section of General Internal Medicine, Wake Forest School of Medicine
| | - Leslie R. Ellis
- Associate Professor, Department of Internal Medicine, Section on Hematology and Oncology, Wake Forest School of Medicine
| |
Collapse
|
26
|
Weber DE, Held JD, Jandarov RA, Kelleher M, Kinnear B, Sall DR, O'Toole JK. Improving trainee clinical documentation through a novel curriculum in internal medicine. J Hosp Med 2022; 17:28-35. [PMID: 35504574 DOI: 10.1002/jhm.27410] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/01/2021] [Revised: 10/21/2021] [Accepted: 12/11/2021] [Indexed: 11/08/2022]
Abstract
BACKGROUND Clinical documentation is a key component of practice. Trainees rarely receive formal training in documentation or assessment of their documentation. Effective methods of improving documentation remain unknown. OBJECTIVE The objective of this study was to determine if the implementation of a documentation curriculum led to improvement in admission note quality. DESIGNS Admission notes written prior to implementation of the curriculum and after the curriculum intervention were assessed. Notes were assessed from two-time frames for both years to account for improvement with time not associated with the intervention. SETTINGS AND PARTICIPANTS Admission notes written by University of Cincinnati interns were assessed. INTERVENTIONS The documentation curriculum consisted of educational sessions and routine admission note assessments with feedback. MAIN OUTCOMES AND MEASURES Admission notes were assessed via the 16 checklist items and two global assessment items of the Admission Note Assessment Tool (ANAT). RESULTS Six ANAT items showed statistically significant differences. The review of systems item improved with the intervention only (odds ratio: 3.61, p < .001) while the assessment and plan item 1 and global assessment item 2 improved with time only (β = .08, p = .03 and β = .25, p = .02, respectively) in univariate models. In univariate models the physical exam item, diagnostic data item 2, and global assessment item 1 showed improvement with both intervention and time, respectively, with additive effects seen in models with both intervention and time. CONCLUSION Several aspects of documentation can improve with a formal documentation curriculum which includes a routine assessment with feedback, and some aspects of documentation improve with time.
Collapse
Affiliation(s)
- Danielle E Weber
- Department of Internal Medicine, University of Cincinnati Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Justin D Held
- Department of Internal Medicine, University of Cincinnati Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Roman A Jandarov
- Department of Internal Medicine, University of Cincinnati Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Matthew Kelleher
- Department of Internal Medicine, University of Cincinnati Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Benjamin Kinnear
- Department of Internal Medicine, University of Cincinnati Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Dana R Sall
- Department of Internal Medicine, HonorHealth Thompson Peak Medical Center, University of Arizona College of Medicine-Phoenix, Scottsdale, Arizona, USA
| | - Jennifer K O'Toole
- Department of Internal Medicine, University of Cincinnati Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
27
|
Quality Improvement: Effect of Standardized Plastic Surgery Consultation Note Templates on Charge Capture. PLASTIC AND RECONSTRUCTIVE SURGERY-GLOBAL OPEN 2021; 9:e3726. [PMID: 34584820 PMCID: PMC8460220 DOI: 10.1097/gox.0000000000003726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2021] [Accepted: 06/10/2021] [Indexed: 11/26/2022]
Abstract
Background: Effective medical record documentation is imperative for both patient care and reimbursement for care provided. The purpose of this study was to compare coding/billing patterns for plastic surgery consultations before and after implementation of a standardized documentation protocol. Methods: Standardized hand, facial trauma, and general plastic surgery consult note templates were created. Following institutional approval, records were reviewed for all plastic surgery consultations from January to October 2019. Template notes were universally implemented in July 2019. Medical coding was performed by a certified professional coder using the 1995 Evaluation and Management Review Worksheet. Coding/billing patterns between groups were compared with and without standardized documentation using univariate analysis. Results: Seventy-five consecutive preimplementation consult notes and 75 consecutive postimplementation consult notes were selected for review. Each group included 25 hand, 25 facial trauma, and 25 general plastic surgery consultation notes. The history and physical examination components of the visit code were more frequently coded as “comprehensive”postimplementation (P = 0.000). There was no significant difference in coding for medical decision making between the two groups (P = 0.340). The final visit code was significantly higher in the postimplementation group (45.3% 99254/99284 versus 2.7%, P = 0.000), and the charges were significantly higher post implementation—average charge per consult $250 versus $203 (P = 0.000) with a 22.8% increase in total charges generated. Conclusion: Utilization of standardized consultation note templates increases the accuracy of coding and associated billing of inpatient and emergency department plastic surgery consultations through documentation and reflection of level of service provided.
Collapse
|
28
|
Vaghani V, Wei L, Mushtaq U, Sittig DF, Bradford A, Singh H. Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments. J Am Med Inform Assoc 2021; 28:2202-2211. [PMID: 34279630 DOI: 10.1093/jamia/ocab121] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2021] [Revised: 05/26/2021] [Accepted: 06/23/2021] [Indexed: 02/05/2023] Open
Abstract
OBJECTIVE Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs). METHODS Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit. RESULTS We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%). CONCLUSION In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.
Collapse
Affiliation(s)
- Viralkumar Vaghani
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Li Wei
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Umair Mushtaq
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Dean F Sittig
- University of Texas-Memorial Hermann Center for Healthcare Quality & Safety, School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Andrea Bradford
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| |
Collapse
|
29
|
Piersa AP, Laiteerapong N, Ham SA, Del Castillo FF, Shah S, Burnet DL, Lee WW. Impact of a medical scribe on clinical efficiency and quality in an academic general internal medicine practice. BMC Health Serv Res 2021; 21:686. [PMID: 34247600 PMCID: PMC8272908 DOI: 10.1186/s12913-021-06710-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Accepted: 06/28/2021] [Indexed: 11/15/2022] Open
Abstract
Background Scribes have been proposed as an intervention to decrease physician electronic health record (EHR) workload and improve clinical quality. We aimed to assess the impact of a scribe on clinical efficiency and quality in an academic internal medicine practice. Methods Six faculty physicians worked with one scribe at an urban academic general internal medicine clinic April through June 2017. Patient visits during the 3 months prior to intervention (baseline, n = 789), unscribed visits during the intervention (concurrent control, n = 605), and scribed visits (n = 579) were included in the study. Clinical efficiency outcomes included time to close encounter, patient time in clinic, and number of visits per clinic session. Quality outcomes included EHR note quality, rates of medication and immunization review, population of patient instructions, reconciliation of outside information, and completion of preventative health recommendations. Results Median time to close encounter (IQR) was lower for scribed visits [0.4 (4.8) days] compared to baseline and unscribed visits [1.2 (5.9) and 2.9 (5.4) days, both p < 0.001]. Scribed notes were more likely to have a clear history of present illness (HPI) [OR = 7.30 (2.35–22.7), p = 0.001] and sufficient HPI information [OR = 2.21 (1.13–4.35), p = 0.02] compared to unscribed notes. Physicians were more likely to review the medication list during scribed vs. baseline visits [OR = 1.70 (1.22–2.35), p = 0.002]. No differences were found in the number of visits per clinic session, patient time in clinic, completion of preventative health recommendations, or other outcomes. Conclusions Working with a scribe in an academic internal medicine practice was associated with more timely documentation. Supplementary Information The online version contains supplementary material available at 10.1186/s12913-021-06710-y.
Collapse
Affiliation(s)
| | | | - Sandra A Ham
- University of Chicago Center for Health and the Social Sciences, Chicago, USA
| | | | - Sachin Shah
- Department of Medicine, University of Chicago, Chicago, IL, USA
| | | | - Wei Wei Lee
- Department of Medicine, University of Chicago, Chicago, IL, USA.
| |
Collapse
|
30
|
Martin DB, Stetson PD, Gilcrease GW, Stillman RC, Sugalski JM, Skinner J, Levy M. Preferences in Oncology History Documentation Styles Among Clinical Practitioners. JCO Oncol Pract 2021; 18:e1-e8. [PMID: 34228492 DOI: 10.1200/op.20.00756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
PURPOSE Clinical notes function as the de facto handoff between providers and assume great importance during unplanned medical encounters. An organized and thorough oncology history is essential in care coordination. We sought to understand reader preferences for oncology history organization by comparing between chronologic and narrative formats. METHODS A convenience sample of 562 clinicians from 19 National Comprehensive Cancer Network Member Institutions responded to a survey comparing two formats of oncology histories, narrative and chronologic, for the same patient. Both histories were consensus-derived real-world examples. Each history was evaluated using semantic differential attributes (thorough, useful, organized, comprehensible, and succinct). Respondents choose a preference between the two styles for history gathering and as the basis of a new note. Open-ended responses were also solicited. RESULTS Respondents preferred the chronologic over the narrative history to prepare for a visit with an unknown patient (66% preference) and as a basis for their own note preparation (77% preference) (P < .01). The chronologic summary was preferred in four of the five measured attributes (useful, organized, comprehensible, and succinct); the narrative summary was favored for thoroughness (P < .01). Open-ended responses reflected the attribute scoring and noted the utility of content describing social determinants of health in the narrative history. CONCLUSION Respondents of this convenience sample preferred a chronologic oncology history to a concise narrative history. Further studies are needed to determine the optimal structure and content of chronologic documentation for oncology patients and the provider effort to use this format.
Collapse
Affiliation(s)
- Daniel B Martin
- Department of Medicine, University of Washington Medical Center, Seattle, WA.,Seattle Cancer Care Alliance, Seattle, WA
| | | | | | - Robert C Stillman
- The Ohio State University Comprehensive Cancer Center, James Cancer Hospital and Solove Research Institute Columbus, OH
| | | | | | - Mia Levy
- Rush University Cancer Center, Chicago, IL
| |
Collapse
|
31
|
Fadahunsi KP, O'Connor S, Akinlua JT, Wark PA, Gallagher J, Carroll C, Car J, Majeed A, O'Donoghue J. Information Quality Frameworks for Digital Health Technologies: Systematic Review. J Med Internet Res 2021; 23:e23479. [PMID: 33835034 PMCID: PMC8167621 DOI: 10.2196/23479] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2020] [Revised: 11/28/2020] [Accepted: 04/03/2021] [Indexed: 12/02/2022] Open
Abstract
Background Digital health technologies (DHTs) generate a large volume of information used in health care for administrative, educational, research, and clinical purposes. The clinical use of digital information for diagnostic, therapeutic, and prognostic purposes has multiple patient safety problems, some of which result from poor information quality (IQ). Objective This systematic review aims to synthesize an IQ framework that could be used to evaluate the extent to which digital health information is fit for clinical purposes. Methods The review was conducted according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines. We searched Embase, MEDLINE, PubMed, CINAHL, Maternity and Infant Care, PsycINFO, Global Health, ProQuest Dissertations and Theses Global, Scopus, and HMIC (the Health Management Information Consortium) from inception until October 2019. Multidimensional IQ frameworks for assessing DHTs used in the clinical context by health care professionals were included. A thematic synthesis approach was used to synthesize the Clinical Information Quality (CLIQ) framework for digital health. Results We identified 10 existing IQ frameworks from which we developed the CLIQ framework for digital health with 13 unique dimensions: accessibility, completeness, portability, security, timeliness, accuracy, interpretability, plausibility, provenance, relevance, conformance, consistency, and maintainability, which were categorized into 3 meaningful categories: availability, informativeness, and usability. Conclusions This systematic review highlights the importance of the IQ of DHTs and its relevance to patient safety. The CLIQ framework for digital health will be useful in evaluating and conceptualizing IQ issues associated with digital health, thus forestalling potential patient safety problems. Trial Registration PROSPERO International Prospective Register of Systematic Reviews CRD42018097142; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=97142 International Registered Report Identifier (IRRID) RR2-10.1136/bmjopen-2018-024722
Collapse
Affiliation(s)
- Kayode Philip Fadahunsi
- Department of Public Health and Primary Care, Imperial College London, London, United Kingdom
| | - Siobhan O'Connor
- School of Health in Social Science, The University of Edinburgh, Edinburgh, United Kingdom
| | - James Tosin Akinlua
- Department of Primary Care and Population Health Sciences, University College London, London, United Kingdom
| | - Petra A Wark
- Department of Primary Care and Population Health Sciences, University College London, London, United Kingdom.,Centre for Intelligent Healthcare, Institute of Health and Wellbeing, Coventry University, Coventry, United Kingdom
| | - Joseph Gallagher
- gHealth Research Group, School of Medicine, University College Dublin, Dublin, Ireland
| | - Christopher Carroll
- Health Economics and Decision Science, School of Health and Related Research, The University of Sheffield, Sheffield, United Kingdom
| | - Josip Car
- Department of Public Health and Primary Care, Imperial College London, London, United Kingdom.,Centre for Population Health Sciences, LKC Medicine, Nayang Technological University, Sungapore, Singapore
| | - Azeem Majeed
- Department of Public Health and Primary Care, Imperial College London, London, United Kingdom
| | - John O'Donoghue
- Malawi eHealth Research Centre, University College Cork, Cork, Ireland.,ASSERT Research Centre, University College Cork, Cork, Ireland
| |
Collapse
|
32
|
Kelly MM, Sklansky DJ, Nackers KAM, Coller RJ, Dean SM, Eickhoff JC, Bentley NL, Nacht CL, Shadman KA. Evaluation and Improvement of Intern Progress Note Assessments and Plans. Hosp Pediatr 2021; 11:401-405. [PMID: 33692085 DOI: 10.1542/hpeds.2020-003244] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
Abstract
OBJECTIVES Progress notes communicate providers' assessments of patients' diagnoses, progress, and treatment plans; however, providers perceive that note quality has degraded since the introduction of electronic health records. In this study, we aimed to (1) develop a tool to evaluate progress note assessments and plans with high interrater reliability and (2) assess whether a bundled intervention was associated with improved intern note quality without delaying note file time. METHODS An 8-member stakeholder team developed a 19-item progress note assessment and plan evaluation (PNAPE) tool and bundled intervention consisting of a new note template and intern training curriculum. Interrater reliability was evaluated by calculating the intraclass correlation coefficient. Blinded assessors then used PNAPE to evaluate assessment and plan quality in pre- and postintervention notes (fall 2017 and 2018). RESULTS PNAPE revealed high internal interrater reliability between assessors (intraclass correlation coefficient = 0.86; 95% confidence interval: 0.66-0.95). Total median PNAPE score increased from 13 (interquartile range [IQR]: 12-15) to 15 (IQR: 14-17; P = .008), and median file time decreased from 4:30 pm (IQR: 2:33 pm-6:20 pm) to 1:13 pm (IQR: 12:05 pm-3:59 pm; P < .001) in pre- and postintervention notes. In the postintervention period, a higher proportion of assessments and plans indicated the primary problem requiring ongoing hospitalization and progress of this problem (P = .0016 and P < .001, respectively). CONCLUSIONS The PNAPE tool revealed high reliability between assessors, and the bundled intervention may be associated with improved intern note assessment and plan quality without delaying file time. Future studies are needed to evaluate whether these improvements can be sustained throughout residency and reproduced in future intern cohorts and other inpatient settings.
Collapse
Affiliation(s)
| | | | | | | | | | - Jens C Eickhoff
- Biostatistics and Medical Informatics, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, Wisconsin; and
| | | | | | | |
Collapse
|
33
|
|
34
|
Epstein JA, Cofrancesco J, Beach MC, Bertram A, Hedian HF, Mixter S, Yeh HC, Berkenblit G. Effect of Outpatient Note Templates on Note Quality: NOTE (Notation Optimization through Template Engineering) Randomized Clinical Trial. J Gen Intern Med 2021; 36:580-584. [PMID: 32901441 PMCID: PMC7947083 DOI: 10.1007/s11606-020-06188-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/16/2020] [Accepted: 08/26/2020] [Indexed: 10/23/2022]
Abstract
BACKGROUND This is the first randomized controlled trial evaluating the impact of note template design on note quality using a simulated patient encounter and a validated assessment tool. OBJECTIVE To compare note quality between two different templates using a novel randomized clinical simulation process. DESIGN A randomized non-blinded controlled trial of a standard note template versus redesigned template. PARTICIPANTS PGY 1-3 IM residents. INTERVENTIONS Residents documented the simulated patient encounter using one of two templates. The standard template was modeled after the usual outpatient progress note. The new template placed the assessment and plan section in the beginning, grouped subjective data into the assessment, and deemphasized less useful elements. MAIN MEASURES Note length; time to note completion; note template evaluation by resident authors; note evaluation by faculty reviewers. KEY RESULTS 36 residents participated, 19 randomized to standard template, 17 to new. New template generated shorter notes (103 vs 285 lines, p < 0.001) that took the same time to complete (19.8 vs 21.6 min, p = 0.654). Using a 5-point Likert scale, residents considered new notes to have increased visual appeal (4 vs 3, p = 0.05) and less redundancy and clutter (4 vs 3, p = 0.006). Overall template satisfaction was not statistically different. Faculty reviewers rated the standard note more up-to-date (4.3 vs 2.7, p = 0.001), accurate (3.9 vs 2.6, p = 0.003), and useful (4 vs 2.8, p = 0.002), but less organized (3.3 vs 4.5, p < 0.001). Total quality was not statistically different. CONCLUSIONS Residents rated the new note template more visually appealing, shorter, and less cluttered. Faculty reviewers rated both note types equivalent in the overall quality but rated new notes inferior in terms of accuracy and usefulness though better organized. This study demonstrates a novel method of a simulated clinical encounter to evaluate note templates before the introduction into practice. TRIAL REGISTRATION ClinicalTrials.gov ID: NCT04333238.
Collapse
Affiliation(s)
- Jeremy A Epstein
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA.
| | - Joseph Cofrancesco
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA
| | - Mary Catherine Beach
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA
| | - Amanda Bertram
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA
| | - Helene F Hedian
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA
| | - Sara Mixter
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA
| | - Hsin-Chieh Yeh
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA
| | - Gail Berkenblit
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Johns Hopkins Outpatient Center, Baltimore, MD, USA
| |
Collapse
|
35
|
Wiebe N, Otero Varela L, Niven DJ, Ronksley PE, Iragorri N, Quan H. Evaluation of interventions to improve inpatient hospital documentation within electronic health records: a systematic review. J Am Med Inform Assoc 2021; 26:1389-1400. [PMID: 31365092 DOI: 10.1093/jamia/ocz081] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2018] [Revised: 04/14/2019] [Accepted: 05/04/2019] [Indexed: 11/14/2022] Open
Abstract
OBJECTIVE Despite the widespread and increasing use of electronic health records (EHRs), the quality of EHRs is problematic. Efforts have been made to address reasons for poor EHR documentation quality. Previous systematic reviews have assessed intervention effectiveness within the outpatient setting or paper documentation. The purpose of this systematic review was to assess the effectiveness of interventions seeking to improve EHR documentation within an inpatient setting. MATERIALS AND METHODS A search strategy was developed based on elaborated inclusion/exclusion criteria. Four databases, gray literature, and reference lists were searched. A REDCap data capture form was used for data extraction, and study quality was assessed using a customized tool. Data were analyzed and synthesized in a narrative, semiquantitative manner. RESULTS Twenty-four studies were included in this systematic review. Owing to high heterogeneity, quantitative comparison was not possible. However, statistically significant results in interventions and affected outcomes were analyzed and discussed. Education and implementation of a new EHR reporting system were the most successful interventions, as evidenced by significantly improved EHR documentation. DISCUSSION Heterogeneity of interventions, outcomes, document type, EHR user, and other variables led to difficulty in measuring EHR documentation quality and effectiveness of interventions. However, the use of education as a primary intervention aligned closely with existing literature in similar fields. CONCLUSIONS Interventions implemented to enhance EHR documentation are highly variable and require standardization. Emphasis should be placed on this novel area of research to improve communication between healthcare providers and facilitate data sharing between centers and countries. PROSPERO Registration Number: CRD42017083494.
Collapse
Affiliation(s)
- Natalie Wiebe
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| | - Lucia Otero Varela
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| | - Daniel J Niven
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada.,Department of Critical Care Medicine, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| | - Paul E Ronksley
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| | - Nicolas Iragorri
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| | - Hude Quan
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| |
Collapse
|
36
|
Nackers KAM, Shadman KA, Kelly MM, Waterman HG, Bentley NL, Gorski DP, Chorney C, Eickhoff JC, Nacht CL, Sklansky DJ. Resident Workshop to Improve Inpatient Documentation Using the Progress Note Assessment and Plan Evaluation (PNAPE) Tool. MEDEDPORTAL : THE JOURNAL OF TEACHING AND LEARNING RESOURCES 2020; 16:11040. [PMID: 33274296 PMCID: PMC7703481 DOI: 10.15766/mep_2374-8265.11040] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/11/2020] [Accepted: 07/23/2020] [Indexed: 06/12/2023]
Abstract
INTRODUCTION Physicians enter residency with varied knowledge regarding the purpose of progress notes and proficiency writing them. The objective of this study was to test whether resident knowledge, beliefs, and confidence writing inpatient progress notes improved after a 2.5-hour workshop intervention. METHODS An educational workshop and note assessment tool was constructed by resident and faculty stakeholders based on a review of literature and institutional best practices. The Progress Note Assessment and Plan Evaluation (PNAPE) tool was designed to assess adherence to best practices in the assessment and plan section of progress notes. Thirty-four residents from a midsized pediatric residency program attended the workshop, which consisted of didactics and small-group work evaluating sample notes using the PNAPE tool. Participants completed a four-question online pre- and postworkshop survey to evaluate their knowledge of progress note components and attitudes regarding note importance. Pre-post analysis was performed with Chi-square testing for true/false questions, and Mann-Whitney testing for Likert scale questions and summative scores. RESULTS A majority of pediatric residents completed the preintervention (n = 26, 76% response rate) and postintervention (n = 23, 68% response rate) surveys. Accurate response rate improved in 15 of 20 of the true/false items, with a statistically significant improvement in five items. Resident perceptions of note importance and confidence in note writing also increased. DISCUSSION A workshop intervention may effectively educate pediatric residents about progress note best practices. Further studies should assess the impact of the intervention on sustained knowledge and beliefs about progress notes and subsequent note quality.
Collapse
Affiliation(s)
- Kirstin A. M. Nackers
- Assistant Professor, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| | - Kristin A. Shadman
- Associate Professor, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| | - Michelle M. Kelly
- Associate Professor, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| | - Helen G. Waterman
- Resident Physician, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| | - Nicole L. Bentley
- Pediatrician, UnityPoint Clinic at Allen Hospital, Waterloo, Iowa, UnityPoint Health
| | - Daniel P. Gorski
- Resident Physician, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| | - Collette Chorney
- Resident Physician, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| | - Jens C. Eickhoff
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison
| | - Carrie L. Nacht
- Research Specialist, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| | - Daniel J. Sklansky
- Associate Professor, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health
| |
Collapse
|
37
|
Polanski WH, Danker A, Zolal A, Senf-Mothes D, Schackert G, Krex D. Improved efficiency of patient admission with electronic health records in neurosurgery. HEALTH INF MANAG J 2020; 51:45-49. [PMID: 32431170 DOI: 10.1177/1833358320920990] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
BACKGROUND Electronic health records (EHRs) may be controversial but they have the potential to improve patient care. We investigated whether the introduction of an electronic template-based admission form for the collection of information about the patient's medical history and neurological and clinical state at admission in the neurosurgical unit might have an impact on the quality of documentation in a discharge record and the amount of time taken to produce this documentation. METHOD A new digital template-based admission form (EHR) was developed and assessed with QNOTE, an assessment tool of medical notes with standardised criteria and the possibility to benchmark the quality of documentations. This was compared to 30 prior paper-based handwritten documentations (HWD) regarding the utilisation of these medical notes for dictation of medical discharge records. RESULTS Implementation of the EHR significantly improved the quality of patient admission documentation with a QNOTE mean grand score of 87 ± 22 (p < 0.0001) compared to prior HWD with 44 ± 30. The mean documentation time for HWD was 8.1 min ± 4.1 min and the dictation time for discharge records was 10.6 min ± 3.5 min. After implementation of EHR, the documentation time increased slightly to 9.6 min ± 2.3 min (n.s.), while the time for dictation of discharge records was reduced to 5.1 min ± 1.2 min (p < 0.0001). There was a clear correlation between a higher quality of documentation and a higher needed documentation time as well as higher quality of documentation and lower dictation times of discharge records. CONCLUSION Implementation of the EHR improved the quality of patient admission documentation and reduced the dictation time of discharge records. IMPLICATIONS It is crucial to involve stakeholders and users of EHRs in a timely manner during the stage of development and implementation phase to ensure optimal results and better usability.
Collapse
Affiliation(s)
| | | | - Amir Zolal
- Technical University of Dresden, Germany
| | | | | | | |
Collapse
|
38
|
Weber DE, Held JD, Jandarov RA, Kelleher M, Kinnear B, Sall D, O'Toole JK. Development and Establishment of Initial Validity Evidence for a Novel Tool for Assessing Trainee Admission Notes. J Gen Intern Med 2020; 35:1078-1083. [PMID: 31993944 PMCID: PMC7174454 DOI: 10.1007/s11606-020-05669-6] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/19/2019] [Accepted: 01/12/2020] [Indexed: 10/25/2022]
Abstract
BACKGROUND Documentation is a key component of practice, yet few curricula have been published to teach trainees proper note construction. Additionally, a gold standard for assessing note quality does not exist, and no documentation assessment tools integrate with established competency-based frameworks. OBJECTIVE To develop and establish initial validity evidence for a novel tool that assesses key components of trainee admission notes and maps to the Accreditation Council for Graduate Medical Education (ACGME) milestone framework. DESIGN Using an iterative, consensus building process we developed the Admission Note Assessment Tool (ANAT). Pilot testing was performed with both the supervising attending and study team raters not involved in care of the patients. The finalized tool was piloted with attendings from other institutions. PARTICIPANTS Local experts participated in tool development and pilot testing. Additional attending physicians participated in pilot testing. MAIN MEASURES Content, response process, and internal structure validity evidence was gathered using Messick's framework. Inter-rater reliability was assessed using percent agreement. KEY RESULTS The final tool consists of 16 checklist items and two global assessment items. Pilot testing demonstrated rater agreement of 72% to 100% for checklist items and 63% to 70% for global assessment items. Note assessment required an average of 12.3 min (SD 3.7). The study generated validity evidence in the domains of content, response process, and internal structure for use of the tool in rating admission notes. CONCLUSIONS The ANAT assesses individual components of a note, incorporates billing criteria, targets note "bloat," allows for narrative feedback, and provides global assessments mapped to the ACGME milestone framework. The ANAT can be used to assess admission notes by any attending and at any time after note completion with minimal rater training. The ANAT allows programs to implement routine note assessment for multiple functions with the use of a single tool.
Collapse
Affiliation(s)
- Danielle E Weber
- Department of Internal Medicine, University of Cincinnati College of Medicine, University of Cincinnati Medical Center, 231 Albert Sabin Way, ML 0535, Cincinnati, OH, 45267, USA. .,Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA.
| | - Justin D Held
- Department of Internal Medicine, University of Cincinnati College of Medicine, University of Cincinnati Medical Center, 231 Albert Sabin Way, ML 0535, Cincinnati, OH, 45267, USA
| | - Roman A Jandarov
- Department of Internal Medicine, University of Cincinnati College of Medicine, University of Cincinnati Medical Center, 231 Albert Sabin Way, ML 0535, Cincinnati, OH, 45267, USA
| | - Matthew Kelleher
- Department of Internal Medicine, University of Cincinnati College of Medicine, University of Cincinnati Medical Center, 231 Albert Sabin Way, ML 0535, Cincinnati, OH, 45267, USA.,Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Ben Kinnear
- Department of Internal Medicine, University of Cincinnati College of Medicine, University of Cincinnati Medical Center, 231 Albert Sabin Way, ML 0535, Cincinnati, OH, 45267, USA.,Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Dana Sall
- Department of Internal Medicine, University of Cincinnati College of Medicine, University of Cincinnati Medical Center, 231 Albert Sabin Way, ML 0535, Cincinnati, OH, 45267, USA
| | - Jennifer K O'Toole
- Department of Internal Medicine, University of Cincinnati College of Medicine, University of Cincinnati Medical Center, 231 Albert Sabin Way, ML 0535, Cincinnati, OH, 45267, USA.,Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| |
Collapse
|
39
|
Pilot trial of semi-automated medical note writing using lexeme hypotheses. Int J Med Inform 2020; 136:104095. [PMID: 32058265 DOI: 10.1016/j.ijmedinf.2020.104095] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2019] [Revised: 11/14/2019] [Accepted: 02/04/2020] [Indexed: 11/24/2022]
Abstract
Clinicians write a billion free text notes per year. These notes are typically replete with errors of all types. No established automated method can extract data from this treasure trove. The practice of medicine therefore remains haphazard and chaotic, resulting in vast economic waste. The lexeme hypotheses are based on our analysis of how records are created. They enable a computer system to predict what issue a clinician will need to address next, based on the environment in which the clinician is working, and what responses the clinician has selected to date. The system uses a lexicon storing the issues (queries) and a range of responses to the issues. When the clinician selects a response, a text fragment is added to the output file. In the first phase of this work, the notes of 69 returning hemophilia patients were scrutinized, and the lexicon was expanded to 847 lexeme queries and 7995 responses to enable the construction of completed notes. The quality of lexeme-generated notes from 20 consecutive subjects was then compared to the clinicians' conventional clinic notes. The system generated grammatically correct notes. In comparison to the traditional clinic note, the lexeme-generated notes were more complete (88 % compared with 62 %), and had less typographical and grammatical errors (0.8 versus 3.5 errors per note). The system notes and traditional notes averaged about 800 words, but the traditional notes had a much wider distribution of lengths. The note-creation rate from marshalling the data to completion using the system averaged 80 wpm, twice as fast as the typical clinician can type. The lexeme method generates more complete, grammatical and organized notes faster than traditional methods. The notes are completely computerized at inception, and they incorporate prompts for clinicians to address otherwise overlooked items. This pilot justifies further exploration of this methodology.
Collapse
|
40
|
Heidemann LA, Heidemann DL, Huey A, Dalton M, Hartley S. Cross-Cover Documentation: Multicenter Development of Assessment Tool for Quality Improvement. TEACHING AND LEARNING IN MEDICINE 2019; 31:519-527. [PMID: 30848962 DOI: 10.1080/10401334.2019.1583567] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Construct: We aimed to develop an assessment tool to measure the quality of electronic health record inpatient documentation of cross-cover events. Background: Cross-cover events occur in hospitalized patients when the primary team is absent. Documentation is critical for safe transitions of care. The quality of documentation for cross-cover events remains unknown, and no standardized tool exists for assessment. Approach: We created an assessment tool for cross-cover note quality with content validation based on input from 15 experts. We measured interrater reliability of the tool and scored cross-cover note quality for hospitalized patients with overnight rapid response team activation on internal medicine services at 2 academic hospitals for 1 year. Patients with a code blue or a clinically insignificant event were excluded. The presence of a note, writer identity (resident or faculty), time from rapid response to documentation, note content (subjective and objective information, diagnosis, and plan), and patient outcomes were compared. Results: The instrument included 8 items to determine quality of cross-cover documentation: reason for physician notification, note written within 6 hours, subjective and objective patient information, diagnosis, treatment, level of care, and whether the attending physician was notified. The mean Cohen's kappa coefficient demonstrated good interrater agreement at 0.76. The instrument was scored in 222 patients with cross-cover notes. Notes documented by faculty scored higher in quality than residents (89% vs. 74% of 8 items present, p < .001). Cross-cover notes often lacked subjective information, diagnosis, and notification of attending, which was present in 60%, 62%, and 7% of notes, respectively. Conclusions: This study presents reliability evidence for an 8-item assessment tool to measure quality of documentation of cross-cover events and indicates improvement is needed for cross-cover education and safe transitions of care in acutely decompensating medical patients.
Collapse
Affiliation(s)
- Lauren A Heidemann
- Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan, USA
| | - Danielle L Heidemann
- Department of Internal Medicine, Henry Ford Health System, Detroit, Michigan, USA
| | - Amanda Huey
- Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan, USA
| | - Melanie Dalton
- Department of Internal Medicine, Henry Ford Health System, Detroit, Michigan, USA
| | - Sarah Hartley
- Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan, USA
| |
Collapse
|
41
|
Otero Varela L, Wiebe N, Niven DJ, Ronksley PE, Iragorri N, Robertson HL, Quan H. Evaluation of interventions to improve electronic health record documentation within the inpatient setting: a protocol for a systematic review. Syst Rev 2019; 8:54. [PMID: 30760323 PMCID: PMC6373133 DOI: 10.1186/s13643-019-0971-2] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/07/2018] [Accepted: 02/04/2019] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Electronic health records (EHRs) are increasing in popularity across national and international healthcare systems. Despite their augmented availability and use, the quality of electronic health records is problematic. There are various reasons for poor documentation quality within the EHR, and efforts have been made to address these areas. Previous systematic reviews have assessed intervention effectiveness within the outpatient setting or within paper documentation. This systematic review aims to assess the effectiveness of different interventions seeking to improve EHR documentation within an inpatient setting. METHODS We will employ a comprehensive search strategy that encompasses four distinct themes: EHR, documentation, interventions, and study design. Four databases (MEDLINE, EMBASE, CENTRAL, and CINAHL) will be searched along with an in-depth examination of the grey literature and reference lists of relevant articles. A customized hybrid study quality assessment tool has been designed, integrating components of the Downs and Black and Newcastle-Ottawa Scales, into a REDCap data capture form to facilitate data extraction and analysis. Given the predicted high heterogeneity between studies, it may not be possible to standardize data for a quantitative comparison and meta-analysis. Thus, data will be synthesized in a narrative, semi-quantitative manner. DISCUSSION This review will summarize the current level of evidence on the effectiveness of interventions implemented to improve inpatient EHR documentation, which could ultimately enhance data quality in administrative health databases. SYSTEMATIC REVIEW REGISTRATION PROSPERO CRD42017083494.
Collapse
Affiliation(s)
- Lucia Otero Varela
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3rd Floor TRW Building, 3280 Hospital Dr. NW, Calgary, AB, T2N 4Z6, Canada.
| | - Natalie Wiebe
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3rd Floor TRW Building, 3280 Hospital Dr. NW, Calgary, AB, T2N 4Z6, Canada
| | - Daniel J Niven
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3rd Floor TRW Building, 3280 Hospital Dr. NW, Calgary, AB, T2N 4Z6, Canada
| | - Paul E Ronksley
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3rd Floor TRW Building, 3280 Hospital Dr. NW, Calgary, AB, T2N 4Z6, Canada
| | - Nicolas Iragorri
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3rd Floor TRW Building, 3280 Hospital Dr. NW, Calgary, AB, T2N 4Z6, Canada
| | - Helen Lee Robertson
- Health Sciences Library, Libraries and Cultural Resources, University of Calgary, Calgary, Canada
| | - Hude Quan
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3rd Floor TRW Building, 3280 Hospital Dr. NW, Calgary, AB, T2N 4Z6, Canada
| |
Collapse
|
42
|
Blažun Vošner H, Železnik D, Kokol P. Bibliometric analysis of the International Medical Informatics Association official journals. Inform Health Soc Care 2018; 44:405-421. [PMID: 30351983 DOI: 10.1080/17538157.2018.1525734] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/28/2022]
Abstract
Objectives: This research article aims to analyze the bibliometric characteristics of four official International Medical Informatics Association (IMIA) journals, namely: the International Journal of Medical Informatics, Methods of Information in Medicine, Applied Clinical Informatics, and Informatics for Health and Social Care.Method: We used descriptive bibliometrics to study the trends of literature production, identify documents` types, most prolific authors, institutions, countries, and most cited publications of all four IMIA journals. Additionally, we visualized the content of published publications using bibliometric mapping to identify journals' main themes and the most prolific and most cited research terms.Results: In total, 6,837 publications were published in all four IMIA journals. Among them, there were 5,137 original articles, meaning that articles were the leading document type. Research is being conducted globally among various research institutions. The most prolific countries are the United States of America, the United Kingdom, Germany, the Netherlands, and Canada. Thematic analyses of clusters show that themes are overlapping between all four journals.Conclusion: The journals contribute to the advances in technology related to health information systems, knowledge-based and decision-making systems, health literacy, and electronic health records.
Collapse
Affiliation(s)
- Helena Blažun Vošner
- Community Healthcare Center Dr. Adolf Drolc Maribor, Department for Science and Research, Maribor, Slovenia
| | - Danica Železnik
- University College of Health Sciences Slovenj Gradec, Research Institute, Slovenj Gradec, Slovenia
| | - Peter Kokol
- University of Maribor, Faculty of Electrical Engineering and Computer Science, Maribor, Slovenia
| |
Collapse
|
43
|
Lybarger KJ, Ostendorf M, Riskin E, Payne TH, White AA, Yetisgen M. Asynchronous Speech Recognition Affects Physician Editing of Notes. Appl Clin Inform 2018; 9:782-790. [PMID: 30332689 PMCID: PMC6192791 DOI: 10.1055/s-0038-1673417] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2018] [Accepted: 08/26/2018] [Indexed: 10/28/2022] Open
Abstract
OBJECTIVE Clinician progress notes are an important record for care and communication, but there is a perception that electronic notes take too long to write and may not accurately reflect the patient encounter, threatening quality of care. Automatic speech recognition (ASR) has the potential to improve clinical documentation process; however, ASR inaccuracy and editing time are barriers to wider use. We hypothesized that automatic text processing technologies could decrease editing time and improve note quality. To inform the development of these technologies, we studied how physicians create clinical notes using ASR and analyzed note content that is revised or added during asynchronous editing. MATERIALS AND METHODS We analyzed a corpus of 649 dictated clinical notes from 9 physicians. Notes were dictated during rounds to portable devices, automatically transcribed, and edited later at the physician's convenience. Comparing ASR transcripts and the final edited notes, we identified the word sequences edited by physicians and categorized the edits by length and content. RESULTS We found that 40% of the words in the final notes were added by physicians while editing: 6% corresponded to short edits associated with error correction and format changes, and 34% were associated with longer edits. Short error correction edits that affect note accuracy are estimated to be less than 3% of the words in the dictated notes. Longer edits primarily involved insertion of material associated with clinical data or assessment and plans. The longer edits improve note completeness; some could be handled with verbalized commands in dictation. CONCLUSION Process interventions to reduce ASR documentation burden, whether related to technology or the dictation/editing workflow, should apply a portfolio of solutions to address all categories of required edits. Improved processes could reduce an important barrier to broader use of ASR by clinicians and improve note quality.
Collapse
Affiliation(s)
- Kevin J. Lybarger
- Department of Electrical Engineering, University of Washington, Seattle, Washington, United States
| | - Mari Ostendorf
- Department of Electrical Engineering, University of Washington, Seattle, Washington, United States
| | - Eve Riskin
- Department of Electrical Engineering, University of Washington, Seattle, Washington, United States
| | - Thomas H. Payne
- Division of General Internal Medicine, University of Washington, Seattle, Washington, United States
| | - Andrew A. White
- Division of General Internal Medicine, University of Washington, Seattle, Washington, United States
| | - Meliha Yetisgen
- Department of Biomedical & Health Informatics, University of Washington, Seattle, Washington, United States
| |
Collapse
|
44
|
Payne TH, Alonso WD, Markiel JA, Lybarger K, Lordon R, Yetisgen M, Zech JM, White AA. Using voice to create inpatient progress notes: effects on note timeliness, quality, and physician satisfaction. JAMIA Open 2018; 1:218-226. [PMID: 31984334 PMCID: PMC6951907 DOI: 10.1093/jamiaopen/ooy036] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2017] [Revised: 08/07/2018] [Accepted: 08/29/2018] [Indexed: 12/31/2022] Open
Abstract
Objectives We describe the evaluation of a system to create hospital progress notes using voice and electronic health record integration to determine if note timeliness, quality, and physician satisfaction are improved. Materials and methods We conducted a randomized controlled trial to measure effects of this new method of writing inpatient progress notes, which evolved over time, on important outcomes. Results Intervention subjects created 709 notes and control subjects created 1143 notes. When adjusting for clustering by provider and secular trends, there was no significant difference between the intervention and control groups in the time between when patients were seen on rounds and when progress notes were viewable by others (95% confidence interval -106.9 to 12.2 min). There were no significant differences in physician satisfaction or note quality between intervention and control. Discussion Though we did not find support for the superiority of this system (Voice-Generated Enhanced Electronic Note System [VGEENS]) for our 3 primary outcomes, if notes are created using voice during or soon after rounds they are available within 10 min. Shortcomings that likely influenced subject satisfaction include the early state of our VGEENS and the short interval for system development before the randomized trial began. Conclusion VGEENS permits voice dictation on rounds to create progress notes and can reduce delay in note availability and may reduce dependence on copy/paste within notes. Timing of dictation determines when notes are available. Capturing notes in near-real-time has potential to apply NLP and decision support sooner than when notes are typed later in the day, and to improve note accuracy.
Collapse
Affiliation(s)
- Thomas H Payne
- Department of Medicine, University of Washington, Seattle, Washington, USA.,UW Medicine Information Technology Services, Seattle, Washington, USA.,Center for Scholarship in Patient Care Quality and Safety, Seattle, Washington, USA
| | - W David Alonso
- Department of Anesthesiology & Pain Medicine, University of Washington, Seattle, Washington, USA
| | - J Andrew Markiel
- UW Medicine Information Technology Services, Seattle, Washington, USA
| | - Kevin Lybarger
- Department of Electrical Engineering, University of Washington, Seattle, Washington, USA
| | - Ross Lordon
- Department of Biomedical Health Informatics and Medical Education, University of Washington, Seattle, Washington, USA
| | - Meliha Yetisgen
- Department of Biomedical Health Informatics and Medical Education, University of Washington, Seattle, Washington, USA
| | - Jennifer M Zech
- Center for Scholarship in Patient Care Quality and Safety, Seattle, Washington, USA
| | - Andrew A White
- Department of Medicine, University of Washington, Seattle, Washington, USA.,UW Medicine Information Technology Services, Seattle, Washington, USA.,Center for Scholarship in Patient Care Quality and Safety, Seattle, Washington, USA
| |
Collapse
|
45
|
Kahn D, Stewart E, Duncan M, Lee E, Simon W, Lee C, Friedman J, Mosher H, Harris K, Bell J, Sharpe B, El-Farra N. A Prescription for Note Bloat: An Effective Progress Note Template. J Hosp Med 2018; 13:378-382. [PMID: 29350222 DOI: 10.12788/jhm.2898] [Citation(s) in RCA: 36] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
Abstract
BACKGROUND United States hospitals have widely adopted electronic health records (EHRs). Despite the potential for EHRs to increase efficiency, there is concern that documentation quality has suffered. OBJECTIVE To examine the impact of an educational session bundled with a progress note template on note quality, length, and timeliness. DESIGN A multicenter, nonrandomized prospective trial. SETTING Four academic hospitals across the United States. PARTICIPANTS Intern physicians on inpatient internal medicine rotations at participating hospitals. INTERVENTION A task force delivered a lecture on current issues with documentation and suggested that interns use a newly designed best practice progress note template when writing daily progress notes. MEASUREMENTS Note quality was rated using a tool designed by the task force comprising a general impression score, the validated Physician Documentation Quality Instrument, 9-item version (PDQI-9), and a competency questionnaire. Reviewers documented number of lines per note and time signed. RESULTS Two hundred preintervention and 199 postintervention notes were collected. Seventy percent of postintervention notes used the template. Significant improvements were seen in the general impression score, all domains of the PDQI-9, and multiple competency items, including documentation of only relevant data, discussion of a discharge plan, and being concise while adequately complete. Notes had approximately 25% fewer lines and were signed on average 1.3 hours earlier in the day. CONCLUSIONS The bundled intervention for progress notes significantly improved the quality, decreased the length, and resulted in earlier note completion across 4 academic medical centers.
Collapse
Affiliation(s)
- Daniel Kahn
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, USA.
| | - Elizabeth Stewart
- Division of Hospital Medicine, Alameda Health System, Oakland, California, USA
| | - Mark Duncan
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, USA
| | - Edward Lee
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, USA
| | - Wendy Simon
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, USA
| | - Clement Lee
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, USA
| | - Jodi Friedman
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, USA
| | - Hilary Mosher
- Department of Internal Medicine, Carver College of Medicine, Iowa City, Iowa, USA
| | - Katherine Harris
- Department of Internal Medicine, Carver College of Medicine, Iowa City, Iowa, USA
| | - John Bell
- Department of Internal Medicine, Division of Hospital Medicine, University of California, San Diego, San Diego, California, USA
| | - Bradley Sharpe
- Department of Medicine, Division of Hospital Medicine, University of California, San Francisco, San Francisco, California, USA
| | - Neveen El-Farra
- Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California, USA
| |
Collapse
|
46
|
Bhise V, Rajan SS, Sittig DF, Vaghani V, Morgan RO, Khanna A, Singh H. Electronic health record reviews to measure diagnostic uncertainty in primary care. J Eval Clin Pract 2018; 24:545-551. [PMID: 29675888 DOI: 10.1111/jep.12912] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/29/2017] [Revised: 02/19/2018] [Accepted: 02/20/2018] [Indexed: 02/05/2023]
Abstract
RATIONALE, AIMS AND OBJECTIVES Diagnostic uncertainty is common in primary care. Because it is challenging to measure, there is inadequate scientific understanding of diagnostic decision-making during uncertainty. Our objective was to understand how diagnostic uncertainty was documented in the electronic health record (EHR) and explore a strategy to retrospectively identify it using clinician documentation. METHODS We reviewed the literature to identify documentation language that could identify both direct expression and indirect inference of diagnostic uncertainty and designed an instrument to facilitate record review. Direct expression included clinician's use of question marks, differential diagnoses, symptoms as diagnosis, or vocabulary such as "probably, maybe, likely, unclear or unknown," while describing the diagnosis. Indirect inference included absence of documented diagnosis at the end of the visit, ordering of multiple consultations or diagnostic tests to resolve diagnostic uncertainty, and use of suspended judgement, test of treatment, and risk-averse disposition. Two physician-reviewers independently reviewed notes on a sample of outpatient visits to identify diagnostic uncertainty at the end of the visit. Documented Ninth Revision of the International Classification of Diseases (ICD-9) diagnosis codes and note quality were assessed. RESULTS Of 389 patient records reviewed, 218 had evidence of diagnostic activity and were included. In 156 visits (71.6%), reviewers identified clinicians who experienced diagnostic uncertainty with moderate inter-reviewer agreement (81.7%; Cohen's kappa: 0.609). Most cases (125, 80.1%) showed evidence of both direct expression and indirect inference. Uncertainty was directly expressed in 139 (89.1%) cases, most commonly by using symptoms as diagnosis (98, 62.8%), and inferred in 144 (92.3%). In more than 1/3 of visits (58, 37.2%), diagnostic uncertainty was recorded inappropriately using ICD-9 codes. CONCLUSIONS While current diagnosis coding mechanisms (ICD-9 and ICD-10) are unable to capture uncertainty, our study finds that review of EHR documentation can help identify diagnostic uncertainty with moderate reliability. Better measurement and understanding of diagnostic uncertainty could help inform strategies to improve the safety and efficiency of diagnosis.
Collapse
Affiliation(s)
- Viraj Bhise
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, TX, USA.,Department of Medicine, Baylor College of Medicine, Houston, TX, USA.,University of Texas School of Public Health, Houston, TX, USA.,John A Burns School of Medicine, University of Hawai'i at Manoa, Honolulu, HI, USA
| | - Suja S Rajan
- University of Texas School of Public Health, Houston, TX, USA
| | - Dean F Sittig
- School of Biomedical Informatics and UT-Memorial Hermann Center for Health Care Quality and Safety, University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Viralkumar Vaghani
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, TX, USA.,Department of Medicine, Baylor College of Medicine, Houston, TX, USA
| | - Robert O Morgan
- University of Texas School of Public Health, Houston, TX, USA
| | - Arushi Khanna
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, TX, USA.,Department of Medicine, Baylor College of Medicine, Houston, TX, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, TX, USA.,Department of Medicine, Baylor College of Medicine, Houston, TX, USA
| |
Collapse
|
47
|
Weir CR, Staes C, Slager S, Taft T, Chidambaram V, Kramer H, Bray BE, Moore SP. What are they trying to do?: An analysis of Action Identities in using electronic documentation in an EHR. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2018; 2017:1764-1772. [PMID: 29854247 PMCID: PMC5977632] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Documentation processes have changed substantially with EHR adoption. User satisfaction studies have focused on usability or cognitive analysis perspectives. Few studies have provided useful information to developers to improve designs. The purpose of this study is to report a 3-pronged approach to deepen understanding of the documentation process, with the intent to provide useful information for future design. This study was conducted in two phases, beginning with cognitive task interviews and observations, followed by post-observation interviews. Twenty-five constructs were identified across the phases, and we observed several patterns of note writing. Participants provided useful information to potentially inform future design. Our study illustrates how electronic documentation serves many clinical processes and is at the core of the medical record. Providers need multiple kinds of notes and ways to display notes. In order to meet provider goals, we must completely re-think the way electronic documentation is composed and displayed.
Collapse
Affiliation(s)
- Charlene R Weir
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah
| | - Catherine Staes
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah
| | - Stacey Slager
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah
| | - Teresa Taft
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah
| | | | - Heidi Kramer
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah
| | - Bruce E Bray
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah
| | - Seneca Perri Moore
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah
| |
Collapse
|
48
|
Al Hadidi S, Upadhaya S, Shastri R, Alamarat Z. Use of dictation as a tool to decrease documentation errors in electronic health records. J Community Hosp Intern Med Perspect 2017; 7:282-286. [PMID: 29147468 PMCID: PMC5676967 DOI: 10.1080/20009666.2017.1379852] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2017] [Accepted: 09/11/2017] [Indexed: 11/21/2022] Open
Abstract
Background: Use of Electronic Health Records is increasing. Copy-and-paste function is frequently used with higher rates of documentation errors. Studies to determine the nature of such errors are needed.Objectives: Determination of the effect of implementing a dictation system for completing notes on the quality of clinical documentation. We hypothesized that implementation of the dictation system for note writing would decrease the rate of errors in the progress notes as well as decrease the rate of copying and pasting. Design/Methods: A prospective interventional study in inpatient medical service for six months’ duration starting in July 2016. Resident physicians’ charts were reviewed by the attending physician on a daily basis. This study was done in a community based hospital affiliated to a university program. Residents’ physicians included Internal Medicine, Transitional year and Combined Internal Medicine Pediatrics residents. Charts reviewed for hospitalized patients. A total of 54 residents were offered a pre-intervention survey indicating their subjective use of copy/paste function. Response rate of 85.18%. Progress notes were reviewed on a daily basis for residents on their inpatient rotation. A total of 621 notes were reviewed. Results: Percentage of notes copied prior to the intervention was 92.73% which decreased to 49.71% post-intervention (RR of 0.54, 95% CI 0.48 0.60 Z statistic 11.005 with p-value <0.0001). Of the copied notes percentage of errors pre-intervention was 58% with no errors identified post-intervention (RR of 0.005, 95% CI 0.0003 0.0795 Z statistic 3.752 with p-value 0.0002). Most of the errors are from notes copied by the same author (85.8%). The most common documentation error was in the physical examination section. Conclusion: Implementing a dictation system eliminated documentation errors over our six months’ study. Further studies are needed to check long effects of using such systems on documentation errors
Collapse
Affiliation(s)
- Samer Al Hadidi
- Hurley Medical Center, Michigan State University, Flint, MI, USA
| | - Sunil Upadhaya
- Internal Medicine Department, Hurley Medical Center/Michigan State University, Flint, MI, USA
| | - Rupal Shastri
- Internal Medicine Department, Rush University, Chicago, USA
| | - Zain Alamarat
- Pediatrics Department, Hurley Medical Center/Michigan State University, Flint, MI, USA
| |
Collapse
|
49
|
Bierman JA, Hufmeyer KK, Liss DT, Weaver AC, Heiman HL. Promoting Responsible Electronic Documentation: Validity Evidence for a Checklist to Assess Progress Notes in the Electronic Health Record. TEACHING AND LEARNING IN MEDICINE 2017; 29:420-432. [PMID: 28497983 DOI: 10.1080/10401334.2017.1303385] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
UNLABELLED Construct: We aimed to develop an instrument to measure the quality of inpatient electronic health record- (EHR-) generated progress notes without requiring raters to review the detailed chart or know the patient. BACKGROUND Notes written in EHRs have generated criticism for being unnecessarily long and redundant, perpetuating inaccuracy and obscuring providers' clinical reasoning. Available assessment tools either focus on outpatient progress notes or require chart review by raters to develop familiarity with the patient. APPROACH We used medical literature, local expert review, and attending focus groups to develop and refine an instrument to evaluate inpatient progress notes. We measured interrater reliability and scored the selected-response elements of the checklist for a sample of 100 progress notes written by PGY-1 trainees on the general medicine service. RESULTS We developed an instrument with 18 selected-response items and four open-ended items to measure the quality of inpatient progress notes written in the EHR. The mean Cohen's kappa coefficient demonstrated good agreement at .67. The mean note score was 66.9% of maximum possible points (SD = 10.6, range = 34.4%-93.3%). CONCLUSIONS We present validity evidence in the domains of content, internal structure, and response process for a new checklist for rating inpatient progress notes. The scored checklist can be completed in approximately 7 minutes by a rater who is not familiar with the patient and can be done without extensive chart review. We further demonstrate that trainee notes show substantial room for improvement.
Collapse
Affiliation(s)
- Jennifer A Bierman
- a Departments of Medicine and Medical Education , Northwestern University Feinberg School of Medicine , Chicago , Illinois , USA
| | - Kathryn Kinner Hufmeyer
- b Department of Medicine , Northwestern University Feinberg School of Medicine , Chicago , Illinois , USA
| | - David T Liss
- b Department of Medicine , Northwestern University Feinberg School of Medicine , Chicago , Illinois , USA
| | - A Charlotta Weaver
- b Department of Medicine , Northwestern University Feinberg School of Medicine , Chicago , Illinois , USA
| | - Heather L Heiman
- c Departments of Medicine and Medical Education , Northwestern University Feinberg School of Medicine , Chicago , Illinois , USA
| |
Collapse
|
50
|
Walker KJ, Wang A, Dunlop W, Rodda H, Ben-Meir M, Staples M. The 9-Item Physician Documentation Quality Instrument (PDQI-9) score is not useful in evaluating EMR (scribe) note quality in Emergency Medicine. Appl Clin Inform 2017; 8:981-993. [PMID: 28956888 PMCID: PMC6220701 DOI: 10.4338/aci2017050080] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2017] [Accepted: 08/02/2017] [Indexed: 11/23/2022] Open
Abstract
BACKGROUND Scribes are assisting Emergency Physicians by writing their electronic clinical notes at the bedside during consultations. They increase physician productivity and improve their working conditions. The quality of Emergency scribe notes is unevaluated and important to determine. OBJECTIVE The primary objective of the study was to determine if the quality of Emergency Department scribe notes was equivalent to physician only notes, using the Physician Documentation Quality Instrument, Nine-item tool (PDQI-9). METHODS This was a retrospective, observational study comparing 110 scribed to 110 non-scribed Emergency Physician notes written at Cabrini Emergency Department, Australia. Consultations during a randomised controlled trial of scribe/doctor productivity in 2016 were used. Emergency physicians and nurses rated randomly selected, blinded and de-identified notes, 2 raters per note. Comparisons were made between paired scribed and unscribed notes and between raters of each note. Characteristics of individual raters were examined. The ability of the tool to discriminate between good and poor notes was tested. RESULTS The PDQI-9 tool has significant issues. Individual items had good internal consistency (Cronbach's alpha=0.93), but there was very poor agreement between raters (Pearson's r=0.07, p=0.270). There were substantial differences in PDQI-9 scores allocated by each rater, with some giving typically lower scores than others, F(25,206)=1.93, p=0.007. The tool was unable to distinguish good from poor notes, F(3,34)=1.15, p=0.342. There was no difference in PDQI-9 score between scribed and non-scribed notes. CONCLUSIONS The PDQI-9 documentation quality tool did not demonstrate reliability or validity in evaluating Emergency Medicine consultation notes. We found no evidence that scribed notes were of poorer quality than non-scribed notes, however Emergency scribe note quality has not yet been determined.
Collapse
Affiliation(s)
- Katherine J Walker
- Dr Katie Walker, Emergency Department Cabrini, 183 Wattletree Rd, Malvern, VIC 3144, Australia, Phone +61 431 272 262,
| | | | | | | | | | | |
Collapse
|