Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)

Accuracy, thoroughness, and quality of outpatient primary care documentation in the U.S. Department of Veterans Affairs

BACKGROUND

Electronic health records (EHRs) can accelerate documentation and may enhance details of notes, or complicate documentation and introduce errors. Comprehensive assessment of documentation quality requires comparing documentation to what transpires during the clinical encounter itself. We assessed outpatient primary care notes and corresponding recorded encounters to determine accuracy, thoroughness, and several additional key measures of documentation quality.

METHODS

Patients and primary care clinicians across five midwestern primary care clinics of the US Department of Veterans Affairs were recruited into a prospective observational study. Clinical encounters were video-recorded and transcribed verbatim. Using the Physician Documentation Quality Instrument (PDQI-9) added to other measures, reviewers scored quality of the documentation by comparing transcripts to corresponding encounter notes. PDQI-9 items were scored from 1 to 5, with higher scores indicating higher quality.

RESULTS

Encounters (N = 49) among 11 clinicians were analyzed. Most issues that patients initiated in discussion were omitted from notes, and nearly half of notes referred to information or observations that could not be verified. Four notes lacked concluding assessments and plans; nine lacked information about when patients should return. Except for thoroughness, PDQI-9 items that were assessed achieved quality scores exceeding 4 of 5 points.

CONCLUSIONS

Among outpatient primary care electronic records examined, most issues that patients initiated in discussion were absent from notes, and nearly half of notes referred to information or observations absent from transcripts. EHRs may contribute to certain kinds of errors. Approaches to improving documentation should consider the roles of the EHR, patient, and clinician together.

Validating the Physician Documentation Quality Instrument for Intensive Care Unit-Ward Transfer Notes

Background

Physician communication failures during transfers of patients from the intensive care unit (ICU) to the general ward are common and can lead to adverse events. Efforts to improve written handoffs during these transfers are increasingly prominent, but no instruments have been developed to assess the quality of physician ICU-ward transfer notes.

Objective

To collect validity evidence for the modified nine-item Physician Documentation Quality Instrument (mPDQI-9) for assessing ICU-ward transfer note usefulness across several hospitals.

Methods

Twenty-four physician raters independently used the mPDQI-9 to grade 12 notes collected from three academic hospitals. A priori, we excluded the "up-to-date" and "accurate" domains, because these could not be assessed without giving the rater access to the complete patient chart. Assessments therefore used the domains "thorough," "useful," "organized," "comprehensible," "succinct," "synthesized," and "consistent." Raters scored each domain on a Likert scale ranging from 1 (low) to 5 (high). The total mPDQI-9 was the sum of these domain scores. The primary outcome was the raters' perceived clinical utility of the notes, and the primary measures of interest were criterion validity (Spearman's ρ) and interrater reliability (intraclass correlation [ICC]).

Results

Mean mPDQI-9 scores by note ranged from 19 (SD = 5.5) to 30 (SD = 4.2). Mean note ratings did not systematically differ by rater expertise (for interaction, P = 0.15). The proportion of raters perceiving each note as independently sufficient for patient care (the primary outcome) ranged from 33% to 100% across the set of notes. We found a moderately positive correlation between mPDQI-9 ratings and raters' overall assessments of each note's clinical utility (ρ = 0.48, P < 0.001). Interrater reliability was strong; the overall ICC was 0.89 (95% confidence interval [CI], 0.80-0.85), and ICCs were similar among reviewer groups. Finally, Cronbach's α was 0.87 (95% CI, 0.84-0.89), indicating good internal consistency.

Conclusions

We report moderate validity evidence for the mPDQI-9 to assess the usefulness of ICU-ward transfer notes written by internal medicine residents.

Off-label prescribing of psychotropics in a psychiatric patient population in Australia

BACKGROUND

This study aimed to investigate the practice of off-label prescribing in both in- and outpatient psychiatry practice.

METHODS

One-hundred inpatient and 100 outpatient medical records from adult patients of an Australian psychiatry service from 2020 to 2021 were examined to determine the prevalence of off-label prescribing as defined by Therapeutic Goods Administration (TGA) indications, adherence to Royal Australian and New Zealand College of Psychiatrists (RANZCP) treatment guidelines, frequency of off-label prescription, and the quality of documentation and informed consent process.

RESULTS

Most prescribing events in both in- and outpatient settings were either on-label or off-label but consistent with RANZCP guidelines. Patients with a schizoaffective disorder diagnosis or displaying aggression were most likely to receive off-label prescriptions. There was no significant difference between in- and outpatient groups in the quality of documentation or consent process.

CONCLUSIONS

In general, off-label prescribing across groups was common, but many decisions were then in line with RANZCP recommendations. That there is a discrepancy between clinical and regulatory bodies has implications for how off-label status is decided.

Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study

BACKGROUND

Medical documentation plays a crucial role in clinical practice, facilitating accurate patient management and communication among health care professionals. However, inaccuracies in medical notes can lead to miscommunication and diagnostic errors. Additionally, the demands of documentation contribute to physician burnout. Although intermediaries like medical scribes and speech recognition software have been used to ease this burden, they have limitations in terms of accuracy and addressing provider-specific metrics. The integration of ambient artificial intelligence (AI)-powered solutions offers a promising way to improve documentation while fitting seamlessly into existing workflows.

OBJECTIVE

This study aims to assess the accuracy and quality of Subjective, Objective, Assessment, and Plan (SOAP) notes generated by ChatGPT-4, an AI model, using established transcripts of History and Physical Examination as the gold standard. We seek to identify potential errors and evaluate the model's performance across different categories.

METHODS

We conducted simulated patient-provider encounters representing various ambulatory specialties and transcribed the audio files. Key reportable elements were identified, and ChatGPT-4 was used to generate SOAP notes based on these transcripts. Three versions of each note were created and compared to the gold standard via chart review; errors generated from the comparison were categorized as omissions, incorrect information, or additions. We compared the accuracy of data elements across versions, transcript length, and data categories. Additionally, we assessed note quality using the Physician Documentation Quality Instrument (PDQI) scoring system.

RESULTS

Although ChatGPT-4 consistently generated SOAP-style notes, there were, on average, 23.6 errors per clinical case, with errors of omission (86%) being the most common, followed by addition errors (10.5%) and inclusion of incorrect facts (3.2%). There was significant variance between replicates of the same case, with only 52.9% of data elements reported correctly across all 3 replicates. The accuracy of data elements varied across cases, with the highest accuracy observed in the "Objective" section. Consequently, the measure of note quality, assessed by PDQI, demonstrated intra- and intercase variance. Finally, the accuracy of ChatGPT-4 was inversely correlated to both the transcript length (P=.05) and the number of scorable data elements (P=.05).

CONCLUSIONS

Our study reveals substantial variability in errors, accuracy, and note quality generated by ChatGPT-4. Errors were not limited to specific sections, and the inconsistency in error types across replicates complicated predictability. Transcript length and data complexity were inversely correlated with note accuracy, raising concerns about the model's effectiveness in handling complex medical cases. The quality and reliability of clinical notes produced by ChatGPT-4 do not meet the standards required for clinical use. Although AI holds promise in health care, caution should be exercised before widespread adoption. Further research is needed to address accuracy, variability, and potential errors. ChatGPT-4, while valuable in various applications, should not be considered a safe alternative to human-generated clinical documentation at this time.

Evaluating Large Language Models for Drafting Emergency Department Discharge Summaries

Importance

Large language models (LLMs) possess a range of capabilities which may be applied to the clinical domain, including text summarization. As ambient artificial intelligence scribes and other LLM-based tools begin to be deployed within healthcare settings, rigorous evaluations of the accuracy of these technologies are urgently needed.

Objective

To investigate the performance of GPT-4 and GPT-3.5-turbo in generating Emergency Department (ED) discharge summaries and evaluate the prevalence and type of errors across each section of the discharge summary.

Design

Cross-sectional study.

Setting

University of California, San Francisco ED.

Participants

We identified all adult ED visits from 2012 to 2023 with an ED clinician note. We randomly selected a sample of 100 ED visits for GPT-summarization.

Exposure

We investigate the potential of two state-of-the-art LLMs, GPT-4 and GPT-3.5-turbo, to summarize the full ED clinician note into a discharge summary.

Main Outcomes and Measures

GPT-3.5-turbo and GPT-4-generated discharge summaries were evaluated by two independent Emergency Medicine physician reviewers across three evaluation criteria: 1) Inaccuracy of GPT-summarized information; 2) Hallucination of information; 3) Omission of relevant clinical information. On identifying each error, reviewers were additionally asked to provide a brief explanation for their reasoning, which was manually classified into subgroups of errors.

Results

From 202,059 eligible ED visits, we randomly sampled 100 for GPT-generated summarization and then expert-driven evaluation. In total, 33% of summaries generated by GPT-4 and 10% of those generated by GPT-3.5-turbo were entirely error-free across all evaluated domains. Summaries generated by GPT-4 were mostly accurate, with inaccuracies found in only 10% of cases, however, 42% of the summaries exhibited hallucinations and 47% omitted clinically relevant information. Inaccuracies and hallucinations were most commonly found in the Plan sections of GPT-generated summaries, while clinical omissions were concentrated in text describing patients' Physical Examination findings or History of Presenting Complaint.

Conclusions and Relevance

In this cross-sectional study of 100 ED encounters, we found that LLMs could generate accurate discharge summaries, but were liable to hallucination and omission of clinically relevant information. A comprehensive understanding of the location and type of errors found in GPT-generated clinical text is important to facilitate clinician review of such content and prevent patient harm.

ChatGPT's Ability to Assist with Clinical Documentation: A Randomized Controlled Trial

INTRODUCTION

Clinical documentation is a critical aspect of health care that enables healthcare providers to communicate effectively with each other and maintain accurate patient care records. Artificial intelligence tools, such as chatbots and virtual assistants, have the potential to assist healthcare providers in clinical documentation. ChatGPT is an artificial intelligence conversational model that generates human-like responses to text-based prompts. In this study, we sought to investigate ChatGPT's ability to assist with writing a history of present illness based on standardized patient histories.

METHODS

A blinded, randomized controlled study was conducted to compare the use of typing, dictation, and ChatGPT as tools to document history of present illness (HPI) of standardized patient histories. Eleven study participants, consisting of medical students, orthopaedic surgery residents, and attending surgeons, completed three HPIs using a different documentation technique for each one. Participants were randomized into cohorts based on the type of documentation technique. Participants were asked to interview standardized patients and document the patient's history of present illness using their assigned method.

RESULTS

ChatGPT was found to be intermediate for speed; dictation was fastest, but produced markedly longer and higher quality patient histories based on Physician Documentation Quality Instrument score compared with dictation and typing. However, ChatGPT included erroneous information in 36% of the documents. Poor agreement existed on the quality of patient histories between reviewers.

DISCUSSION

Our study suggests that ChatGPT has the potential to improve clinical documentation by producing more comprehensive and organized HPIs. ChatGPT can generate longer and more detailed documentation compared with typing or dictation documentation methods. However, additional studies are needed to investigate and address concerns regarding privacy, bias, and accuracy of information.

Medical Student Note Quality on a Pediatrics Core Clerkship Differs by Service

Introduction Medical students rotate on various clinical disciplines with the same professional goal of learning medical documentation. This study investigated differences between medical student notes on inpatient general and subspecialty pediatric services by comparing note quality, length, and file time. Methods In a single-site, observational cohort study, medical students in the Core Clerkship in Pediatrics (CCP) from July 2020 to June 2021 participated in a note-writing didactic course. We compared notes from medical students completing their inpatient assignment on a general pediatric service to those who completed it on a pediatric subspecialty service. Primary outcomes were note quality measured by Physician Documentation Quality Instrument-9 (PDQI9), note length (measured by line count), and file time (measured by hours to completion since 6 AM on the morning of note initiation). Results We evaluated 84 notes from 84 medical students on the general pediatric services and 50 notes from 49 medical students on the pediatric subspecialty services. Note quality measured by PDQI9 was significantly higher for general pediatric service notes compared to pediatric subspecialty service notes (p = 0.03). General pediatric service notes were significantly shorter (p < 0.001). We found no difference in file time (p = 0.23). Conclusion Medical student notes on pediatric subspecialty services scored significantly lower in quality and were longer compared to general pediatric services, demonstrating the need for a more tailored note-writing curriculum and note template based on service.

Assessing the Impact of a Note-Writing Session and Standardized Note Template on Medical Student Note Length and Quality

PURPOSE

To determine whether a formal note-writing session and note template for medical students (MS) during the Core Clerkship in Pediatrics (CCP) increase note quality, shortens note length, and decreases time of documentation.

METHODS

In this single site, prospective study, MS participating in an 8-week CCP received a didactic session on note-writing in the electronic health record (EHR) and utilized EHR template developed for the study. We assessed note quality (measured by Physician Documentation Quality Instrument-9 [PDQI-9]), note length and note documentation time in this group compared to MS notes on the CCP in the prior academic year. We used descriptive statistics and Kruskal-Wallis tests for analysis.

RESULTS

We analyzed 121 notes written by 40 students in the control group and 92 notes writing by 41 students in the intervention group. Notes from the intervention group were more "up to date," "accurate," "organized," and "comprehensible" compared to the control group (P = 0.02, P = 0.04, P = 0.01, and P = 0.02, respectively). Intervention group notes received higher cumulative PDQI-9 scores compared to the control group (median score 38 (IQR 34-42) versus 36 (IQR 32-40) out of 45 total, P = 0.04). Intervention group notes were approximately 35% shorter than the control group notes (median 68.5 lines vs 105 lines, P < 0.0001) and were signed earlier than control group notes (median file time 316 minute vs 352 minute, P = 0.02).

CONCLUSIONS

The intervention successfully decreased note length, improved note quality based on standardized metrics, and reduced time to completion of note documentation.

Assessing the Assessment-Developing and Deploying a Novel Tool for Evaluating Clinical Notes' Diagnostic Assessment Quality

BACKGROUND

Ambulatory diagnostic errors are increasingly being recognized as an important quality and safety issue, and while measures of diagnostic quality have been sought, tools to evaluate diagnostic assessments in the medical record are lacking.

OBJECTIVE

To develop and test a tool to measure diagnostic assessment note quality in primary care urgent encounters and identify common elements and areas for improvement in diagnostic assessment.

DESIGN

Retrospective chart review of urgent care encounters at an urban academic setting.

PARTICIPANTS

Primary care physicians.

MAIN MEASURES

The Assessing the Assessment (ATA) instrument was evaluated for inter-rater reliability, internal consistency, and findings from its application to EHR notes.

KEY RESULTS

ATA had reasonable performance characteristics (kappa 0.63, overall Cronbach's alpha 0.76). Variability in diagnostic assessment was seen in several domains. Two components of situational awareness tended to be well-documented ("Don't miss diagnoses" present in 84% of charts, red flag symptoms in 87%), while Psychosocial context was present only 18% of the time.

CONCLUSIONS

The ATA tool is a promising framework for assessing and identifying areas for improvement in diagnostic assessments documented in clinical encounters.

Your note, your way: how to write an inpatient progress note accurately and efficiently as an intern

A physician's progress note is an essential piece of documentation regarding key events and the daily status of patients during their hospital stay. It serves not only as a communication tool between care team members, but also chronicles clinical status and pertinent updates to their medical care. Despite the importance of these documents, little literature exists on how to help residents to improve the quality of their daily progress notes. A narrative literature review of English language literature was performed and summated to provide recommendations on how to write an inpatient progress note more accurately and efficiently. In addition, the authors will also introduce a method to build a personal template with the goal of extracting relevant data automatically to reduce clicks for an inpatient progress note in the electronic medical record system.

How does work environment relate to diagnostic quality? A prospective, mixed methods study in primary care

OBJECTIVES

The quest to measure and improve diagnosis has proven challenging; new approaches are needed to better understand and measure key elements of the diagnostic process in clinical encounters. The aim of this study was to develop a tool assessing key elements of the diagnostic assessment process and apply it to a series of diagnostic encounters examining clinical notes and encounters' recorded transcripts. Additionally, we aimed to correlate and contextualise these findings with measures of encounter time and physician burnout.

DESIGN

We audio-recorded encounters, reviewed their transcripts and associated them with their clinical notes and findings were correlated with concurrent Mini Z Worklife measures and physician burnout.

SETTING

Three primary urgent-care settings.

PARTICIPANTS

We conducted in-depth evaluations of 28 clinical encounters delivered by seven physicians.

RESULTS

Comparing encounter transcripts with clinical notes, in 24 of 28 (86%) there was high note/transcript concordance for the diagnostic elements on our tool. Reliably included elements were red flags (92% of notes/encounters), aetiologies (88%), likelihood/uncertainties (71%) and follow-up contingencies (71%), whereas psychosocial/contextual information (35%) and mentioning common pitfalls (7%) were often missing. In 22% of encounters, follow-up contingencies were in the note, but absent from the recorded encounter. There was a trend for higher burnout scores being associated with physicians less likely to address key diagnosis items, such as psychosocial history/context.

CONCLUSIONS

A new tool shows promise as a means of assessing key elements of diagnostic quality in clinical encounters. Work conditions and physician reactions appear to correlate with diagnostic behaviours. Future research should continue to assess relationships between time pressure and diagnostic quality.

Novel Note Templates to Enhance Signal and Reduce Noise in Medical Documentation: a Prospective Improvement Study

BACKGROUND

The introduction of electronic workflows has allowed for the flow of raw un-contextualized clinical data into medical documentation. As a result, many electronic notes have become replete of "noise" and deplete of clinically significant "signals". There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating.

OBJECTIVE

To describe the development and impact of a novel set of templates designed to change the flow of information in medical documentation.

METHODS

This is a multi-hospital nonrandomized prospective improvement study conducted on the Inpatient General Internal Medicine Service across three hospital campuses at the New York University (NYU) Langone Health System. A group of physician leaders representing each campus met biweekly for six months. The output of these meetings included 1) a conceptualization of the note bloat problem as a dysfunction in information flow 2) a set of guiding principles for organizational documentation improvement 3) the design and build of novel electronic templates that reduced the flow of extraneous information into provider notes by providing link outs to best practice data visualizations and 4) a documentation improvement curriculum for inpatient medicine providers. Prior to go-live, pragmatic usability testing was performed with the new progress note template, and the overall user experience measured using the System Usability Scale (SUS). Primary outcomes measures after go-live include template utilization rate and note length in characters.

RESULTS

In usability testing amongst 22 medicine providers, the new progress note template averaged a usability score of 90.6/100 on the System Usability Scale. 77% of providers strongly agreed that the new template was easy to use. 68% strongly agreed that they would like to use the template frequently. In the three months after template implementation, General Internal Medicine providers wrote 65% of all inpatient notes with the new templates. During this period of time the organization saw a 46%, 47%, and 32% reduction in note length for general medicine progress notes, consults, and H&Ps, respectively, when compared to a baseline measurement period prior to interventions.

CONCLUSIONS

A bundled intervention that included deployment of novel templates for inpatient general medicine providers significantly reduced average note length on the clinical service. Templates designed to reduce the flow of extraneous information into provider notes performed well during usability testing, and these templates were rapidly adopted across all hospital campuses. Further research is needed to assess the impact of novel templates on note quality, provider efficiency and patient outcomes.

CLINICALTRIAL

Comparison of clinical note quality between an automated digital intake tool and the standard note in the emergency department

BACKGROUND

Medical encounters require an efficient and focused history of present illness (HPI) to create differential diagnoses and guide diagnostic testing and treatment. Our aim was to compare the HPI of notes created by an automated digital intake tool versus standard medical notes created by clinicians.

METHODS

Prospective trial in a quaternary academic Emergency Department (ED). Notes were compared using the 5-point Physician Documentation Quality Instrument (PDQI-9) scale and the Centers for Medicare & Medicaid Services (CMS) level of complexity index. Reviewers were board certified emergency medicine physicians blinded to note origin. Reviewers received training and calibration prior to note assessments. A difference of 1 point was considered clinically significant. Analysis included McNemar's (binary), Wilcoxon-rank (Likert), and agreement with Cohen's Kappa.

RESULTS

A total of 148 ED medical encounters were charted by both digital note and standard clinical note. The ability to capture patient information was assessed through comparison of note content across paired charts (digital-standard note on the same patient), as well as scores given by the reviewers. Reviewer agreement was kappa 0.56 (CI 0.49-0.64), indicating moderate level of agreement between reviewers scoring the same patient chart. Considering all 18 questions across PDQI-9 and CMS scales, the average agreement between standard clinical note and digital note was 54.3% (IQR 44.4-66.7%). There was a moderate level of agreement between content of standard and digital notes (kappa 0.54, 95%CI 0.49-0.60). The quality of the digital note was within the 1 point clinically significant difference for all of the attributes, except for conciseness. Digital notes had a higher frequency of CMS severity elements identified.

CONCLUSION

Digitally generated clinical notes had moderate agreement compared to standard clinical notes and within the one point clinically significant difference except for the conciseness attribute. Digital notes more reliably documented billing components of severity. The use of automated notes should be further explored to evaluate its utility in facilitating documentation of patient encounters.

Assessment of Clinical Information Quality in Digital Health Technologies: International eDelphi Study

BACKGROUND

Digital health technologies (DHTs), such as electronic health records and prescribing systems, are transforming health care delivery around the world. The quality of information in DHTs is key to the quality and safety of care. We developed a novel clinical information quality (CLIQ) framework to assess the quality of clinical information in DHTs.

OBJECTIVE

This study explored clinicians' perspectives on the relevance, definition, and assessment of information quality dimensions in the CLIQ framework.

METHODS

We used a systematic and iterative eDelphi approach to engage clinicians who had information governance roles or personal interest in information governance; the clinicians were recruited through purposive and snowball sampling techniques. Data were collected using semistructured online questionnaires until consensus was reached on the information quality dimensions in the CLIQ framework. Responses on the relevance of the dimensions were summarized to inform decisions on retention of the dimensions according to prespecified rules. Thematic analysis of the free-text responses was used to revise definitions and the assessment of dimensions.

RESULTS

Thirty-five clinicians from 10 countries participated in the study, which was concluded after the second round. Consensus was reached on all dimensions and categories in the CLIQ framework: informativeness (accuracy, completeness, interpretability, plausibility, provenance, and relevance), availability (accessibility, portability, security, and timeliness), and usability (conformance, consistency, and maintainability). A new dimension, searchability, was introduced in the availability category to account for the ease of finding needed information in the DHTs. Certain dimensions were renamed, and some definitions were rephrased to improve clarity.

CONCLUSIONS

The CLIQ framework reached a high expert consensus and clarity of language relating to the information quality dimensions. The framework can be used by health care managers and institutions as a pragmatic tool for identifying and forestalling information quality problems that could compromise patient safety and quality of care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1136/bmjopen-2021-057430.

Writing Is Thinking: Implementation and Evaluation of an Internal Medicine Residency Clinical Reasoning and Documentation Curriculum

With increasingly complicated patients and faster throughput, time for thorough critical thinking and thoughtful clinical documentation is limited, especially in the training environment. Advocating for the value of clinical documentation as a robust opportunity for critical thinking, we describe the implementation and evaluation of a clinical reasoning and documentation curriculum for internal medicine residents. Our curriculum employed facilitated discussion, practical application, and a resident-as-teacher model. Resident surveys showed improved perceptions of the clinical and educational value of clinical documentation. Residents reported increased feedback to interns about their documentation and more appreciation of documentation as a venue for critical thinking.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-022-01570-5.

Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning Documentation

BACKGROUND

Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment.

OBJECTIVE

The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes.

DESIGN, PARTICIPANTS, MAIN MEASURES

Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend.

KEY RESULTS

The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications).

CONCLUSIONS

The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.

Improving Medical Student Inpatient Documentation Through Feedback Using a Note Assessment Tool

Introduction

Documentation within the Electronic Health Record (EHR) is an essential skill for medical students to succeed in residency and post-residency training. The increased use of medical student progress notes for billable services raises the need for the education and assessment of quality note writing. We hypothesized that structured note feedback using a note assessment tool would improve the quality of medical student inpatient progress notes.

Methods

We conducted a retrospective study to review the quality of student inpatient progress notes written before and after structured feedback using the Responsible Electronic Documentation (RED) checklist throughout a third-year internal medicine clerkship. The first intervention group received feedback from clerkship directors in the 2017-2018 academic year and the second intervention group received feedback from ward residents/attendings in the 2018-2019 academic year. Within each intervention group, the total note scores from pre and post-intervention were compared.

Results

Feedback from clerkship directors yielded a greater increase in students’ total note score from pre to post-intervention compared to ward resident/attending feedback (F(1,255) = 12.84, p < 0.001). Cohen’s d effect size value was greater for the clerkship director feedback arm (d=0.71) compared to the ward resident/attending feedback arm (d=0.24). Post-hoc analyses using dependent sample t-tests revealed that there were significant increases in total note scores from pre to post-intervention for both the clerkship director arm (t(123) = 8.26, p < 0.001, d = 0.71) and the ward resident/attending arm (t(132) = 2.85, p = 0.005, d = 0.24).

Conclusion

Clerkship director feedback led to a greater increase in medical student documentation compared to ward attending/resident feedback. Nonetheless, structured feedback with a note assessment tool, whether from clerkship directors or ward attendings/residents, leads to a significant improvement in medical student documentation. Though there are various methods for providing feedback, educators can use the RED checklist to provide clear guidelines that will facilitate note-writing feedback.

A Practical Approach for Monitoring the Use of Copy-Paste in Clinical Notes

The use of copy-paste in authoring clinical notes has been widely embraced by busy providers, but inappropriate copy-paste has been lambasted by critics for introducing risks related to patient safety and regulatory compliance. At an integrated academic health system with over 4,100 providers writing notes, we developed a pragmatic approach to assess the use of copy-paste. From January 1-December 31, 2020, approximately 2.3M inpatient notes and 6.6M ambulatory clinic notes were authored in our electronic health record. Of the inpatient notes, 42% used copy-paste, and 19% of overall note content was copied; in ambulatory notes, 18% used copy-paste and 12% of note content was copied. We describe an approach for including providers' copy-paste usage statistics into the ongoing professional practice evaluation process required for hospital accreditation, thereby offering individual training opportunities related to the lack of use of copy-paste or its potential overuse.

Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback

BACKGROUND

Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability.

OBJECTIVE

Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool.

DESIGN, PARTICIPANTS, AND MAIN MEASURES

The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting.

KEY RESULTS

The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality.

CONCLUSIONS

The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.

Integrating the Electronic Health Record Into Patient Encounters: An Introductory Standardized Patient Exercise for Preclinical Medical Students

INTRODUCTION

Increasingly, use of the electronic health record (EHR) is interwoven into even the most basic patient care tasks. Accordingly, learning how to utilize the EHR during patient encounters is important for medical students as they develop their clinical skills. Existing EHR curricula have focused primarily on doctor-patient relationship skills. We developed a session for our preclinical students on EHR-related doctor-patient relationship skills as well as on using the EHR to verify data and focus one's history taking.

METHODS

We developed student notes, three training videos, four standardized patient (SP) cases, and a simplified, simulated EHR based on these cases. Students reviewed the notes and videos prior to class. During class, students practiced EHR-related communication and data-collection strategies by interviewing an SP while interacting with the simulated EHR. Following each encounter, students received feedback from a small group of peers and faculty.

RESULTS

Two-hundred eighty-nine second-year medical students participated this session in 2019 and 2020, and 27 (19%, 2019) and 40 (28%, 2020) students, respectively, completed the postsession evaluation. Most respondents rated the SP activity as extremely or quite effective for practicing doctor-patient relationship strategies while interacting with the EHR (89%, 2019; 83%, 2020) and for practicing verification of EHR data during a patient encounter (81%, 2019; 86%, 2020).

DISCUSSION

This training session was effective for introducing preclinical medical students to fundamental concepts and skills related to incorporating the EHR into patient encounters and offers a low-cost approach to teaching early medical students these important skills.

Improving trainee clinical documentation through a novel curriculum in internal medicine

BACKGROUND

Clinical documentation is a key component of practice. Trainees rarely receive formal training in documentation or assessment of their documentation. Effective methods of improving documentation remain unknown.

OBJECTIVE

The objective of this study was to determine if the implementation of a documentation curriculum led to improvement in admission note quality.

DESIGNS

Admission notes written prior to implementation of the curriculum and after the curriculum intervention were assessed. Notes were assessed from two-time frames for both years to account for improvement with time not associated with the intervention.

SETTINGS AND PARTICIPANTS

Admission notes written by University of Cincinnati interns were assessed.

INTERVENTIONS

The documentation curriculum consisted of educational sessions and routine admission note assessments with feedback.

MAIN OUTCOMES AND MEASURES

Admission notes were assessed via the 16 checklist items and two global assessment items of the Admission Note Assessment Tool (ANAT).

RESULTS

Six ANAT items showed statistically significant differences. The review of systems item improved with the intervention only (odds ratio: 3.61, p < .001) while the assessment and plan item 1 and global assessment item 2 improved with time only (β = .08, p = .03 and β = .25, p = .02, respectively) in univariate models. In univariate models the physical exam item, diagnostic data item 2, and global assessment item 1 showed improvement with both intervention and time, respectively, with additive effects seen in models with both intervention and time.

CONCLUSION

Several aspects of documentation can improve with a formal documentation curriculum which includes a routine assessment with feedback, and some aspects of documentation improve with time.

Quality Improvement: Effect of Standardized Plastic Surgery Consultation Note Templates on Charge Capture

Background:

Effective medical record documentation is imperative for both patient care and reimbursement for care provided. The purpose of this study was to compare coding/billing patterns for plastic surgery consultations before and after implementation of a standardized documentation protocol.

Methods:

Standardized hand, facial trauma, and general plastic surgery consult note templates were created. Following institutional approval, records were reviewed for all plastic surgery consultations from January to October 2019. Template notes were universally implemented in July 2019. Medical coding was performed by a certified professional coder using the 1995 Evaluation and Management Review Worksheet. Coding/billing patterns between groups were compared with and without standardized documentation using univariate analysis.

Results:

Seventy-five consecutive preimplementation consult notes and 75 consecutive postimplementation consult notes were selected for review. Each group included 25 hand, 25 facial trauma, and 25 general plastic surgery consultation notes. The history and physical examination components of the visit code were more frequently coded as “comprehensive”postimplementation (P = 0.000). There was no significant difference in coding for medical decision making between the two groups (P = 0.340). The final visit code was significantly higher in the postimplementation group (45.3% 99254/99284 versus 2.7%, P = 0.000), and the charges were significantly higher post implementation—average charge per consult $250 versus $203 (P = 0.000) with a 22.8% increase in total charges generated.

Conclusion:

Utilization of standardized consultation note templates increases the accuracy of coding and associated billing of inpatient and emergency department plastic surgery consultations through documentation and reflection of level of service provided.

Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments

OBJECTIVE

Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs).

METHODS

Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit.

RESULTS

We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%).

CONCLUSION

In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.

Impact of a medical scribe on clinical efficiency and quality in an academic general internal medicine practice

Background

Scribes have been proposed as an intervention to decrease physician electronic health record (EHR) workload and improve clinical quality. We aimed to assess the impact of a scribe on clinical efficiency and quality in an academic internal medicine practice.

Methods

Six faculty physicians worked with one scribe at an urban academic general internal medicine clinic April through June 2017. Patient visits during the 3 months prior to intervention (baseline, n = 789), unscribed visits during the intervention (concurrent control, n = 605), and scribed visits (n = 579) were included in the study. Clinical efficiency outcomes included time to close encounter, patient time in clinic, and number of visits per clinic session. Quality outcomes included EHR note quality, rates of medication and immunization review, population of patient instructions, reconciliation of outside information, and completion of preventative health recommendations.

Results

Median time to close encounter (IQR) was lower for scribed visits [0.4 (4.8) days] compared to baseline and unscribed visits [1.2 (5.9) and 2.9 (5.4) days, both p < 0.001]. Scribed notes were more likely to have a clear history of present illness (HPI) [OR = 7.30 (2.35–22.7), p = 0.001] and sufficient HPI information [OR = 2.21 (1.13–4.35), p = 0.02] compared to unscribed notes. Physicians were more likely to review the medication list during scribed vs. baseline visits [OR = 1.70 (1.22–2.35), p = 0.002]. No differences were found in the number of visits per clinic session, patient time in clinic, completion of preventative health recommendations, or other outcomes.

Conclusions

Working with a scribe in an academic internal medicine practice was associated with more timely documentation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06710-y.

Preferences in Oncology History Documentation Styles Among Clinical Practitioners

PURPOSE

Clinical notes function as the de facto handoff between providers and assume great importance during unplanned medical encounters. An organized and thorough oncology history is essential in care coordination. We sought to understand reader preferences for oncology history organization by comparing between chronologic and narrative formats.

METHODS

A convenience sample of 562 clinicians from 19 National Comprehensive Cancer Network Member Institutions responded to a survey comparing two formats of oncology histories, narrative and chronologic, for the same patient. Both histories were consensus-derived real-world examples. Each history was evaluated using semantic differential attributes (thorough, useful, organized, comprehensible, and succinct). Respondents choose a preference between the two styles for history gathering and as the basis of a new note. Open-ended responses were also solicited.

RESULTS

Respondents preferred the chronologic over the narrative history to prepare for a visit with an unknown patient (66% preference) and as a basis for their own note preparation (77% preference) (P < .01). The chronologic summary was preferred in four of the five measured attributes (useful, organized, comprehensible, and succinct); the narrative summary was favored for thoroughness (P < .01). Open-ended responses reflected the attribute scoring and noted the utility of content describing social determinants of health in the narrative history.

CONCLUSION

Respondents of this convenience sample preferred a chronologic oncology history to a concise narrative history. Further studies are needed to determine the optimal structure and content of chronologic documentation for oncology patients and the provider effort to use this format.

Information Quality Frameworks for Digital Health Technologies: Systematic Review

Background

Digital health technologies (DHTs) generate a large volume of information used in health care for administrative, educational, research, and clinical purposes. The clinical use of digital information for diagnostic, therapeutic, and prognostic purposes has multiple patient safety problems, some of which result from poor information quality (IQ).

Objective

This systematic review aims to synthesize an IQ framework that could be used to evaluate the extent to which digital health information is fit for clinical purposes.

Methods

The review was conducted according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines. We searched Embase, MEDLINE, PubMed, CINAHL, Maternity and Infant Care, PsycINFO, Global Health, ProQuest Dissertations and Theses Global, Scopus, and HMIC (the Health Management Information Consortium) from inception until October 2019. Multidimensional IQ frameworks for assessing DHTs used in the clinical context by health care professionals were included. A thematic synthesis approach was used to synthesize the Clinical Information Quality (CLIQ) framework for digital health.

Results

We identified 10 existing IQ frameworks from which we developed the CLIQ framework for digital health with 13 unique dimensions: accessibility, completeness, portability, security, timeliness, accuracy, interpretability, plausibility, provenance, relevance, conformance, consistency, and maintainability, which were categorized into 3 meaningful categories: availability, informativeness, and usability.

Conclusions

This systematic review highlights the importance of the IQ of DHTs and its relevance to patient safety. The CLIQ framework for digital health will be useful in evaluating and conceptualizing IQ issues associated with digital health, thus forestalling potential patient safety problems.

Trial Registration

PROSPERO International Prospective Register of Systematic Reviews CRD42018097142; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=97142

International Registered Report Identifier (IRRID)

RR2-10.1136/bmjopen-2018-024722

Evaluation and Improvement of Intern Progress Note Assessments and Plans

OBJECTIVES

Progress notes communicate providers' assessments of patients' diagnoses, progress, and treatment plans; however, providers perceive that note quality has degraded since the introduction of electronic health records. In this study, we aimed to (1) develop a tool to evaluate progress note assessments and plans with high interrater reliability and (2) assess whether a bundled intervention was associated with improved intern note quality without delaying note file time.

METHODS

An 8-member stakeholder team developed a 19-item progress note assessment and plan evaluation (PNAPE) tool and bundled intervention consisting of a new note template and intern training curriculum. Interrater reliability was evaluated by calculating the intraclass correlation coefficient. Blinded assessors then used PNAPE to evaluate assessment and plan quality in pre- and postintervention notes (fall 2017 and 2018).

RESULTS

PNAPE revealed high internal interrater reliability between assessors (intraclass correlation coefficient = 0.86; 95% confidence interval: 0.66-0.95). Total median PNAPE score increased from 13 (interquartile range [IQR]: 12-15) to 15 (IQR: 14-17; P = .008), and median file time decreased from 4:30 pm (IQR: 2:33 pm-6:20 pm) to 1:13 pm (IQR: 12:05 pm-3:59 pm; P < .001) in pre- and postintervention notes. In the postintervention period, a higher proportion of assessments and plans indicated the primary problem requiring ongoing hospitalization and progress of this problem (P = .0016 and P < .001, respectively).

CONCLUSIONS

The PNAPE tool revealed high reliability between assessors, and the bundled intervention may be associated with improved intern note assessment and plan quality without delaying file time. Future studies are needed to evaluate whether these improvements can be sustained throughout residency and reproduced in future intern cohorts and other inpatient settings.

Effect of Outpatient Note Templates on Note Quality: NOTE (Notation Optimization through Template Engineering) Randomized Clinical Trial

BACKGROUND

This is the first randomized controlled trial evaluating the impact of note template design on note quality using a simulated patient encounter and a validated assessment tool.

OBJECTIVE

To compare note quality between two different templates using a novel randomized clinical simulation process.

DESIGN

A randomized non-blinded controlled trial of a standard note template versus redesigned template.

PARTICIPANTS

PGY 1-3 IM residents.

INTERVENTIONS

Residents documented the simulated patient encounter using one of two templates. The standard template was modeled after the usual outpatient progress note. The new template placed the assessment and plan section in the beginning, grouped subjective data into the assessment, and deemphasized less useful elements.

MAIN MEASURES

Note length; time to note completion; note template evaluation by resident authors; note evaluation by faculty reviewers.

KEY RESULTS

36 residents participated, 19 randomized to standard template, 17 to new. New template generated shorter notes (103 vs 285 lines, p < 0.001) that took the same time to complete (19.8 vs 21.6 min, p = 0.654). Using a 5-point Likert scale, residents considered new notes to have increased visual appeal (4 vs 3, p = 0.05) and less redundancy and clutter (4 vs 3, p = 0.006). Overall template satisfaction was not statistically different. Faculty reviewers rated the standard note more up-to-date (4.3 vs 2.7, p = 0.001), accurate (3.9 vs 2.6, p = 0.003), and useful (4 vs 2.8, p = 0.002), but less organized (3.3 vs 4.5, p < 0.001). Total quality was not statistically different.

CONCLUSIONS

Residents rated the new note template more visually appealing, shorter, and less cluttered. Faculty reviewers rated both note types equivalent in the overall quality but rated new notes inferior in terms of accuracy and usefulness though better organized. This study demonstrates a novel method of a simulated clinical encounter to evaluate note templates before the introduction into practice.

TRIAL REGISTRATION

ClinicalTrials.gov ID: NCT04333238.

Evaluation of interventions to improve inpatient hospital documentation within electronic health records: a systematic review

OBJECTIVE

Despite the widespread and increasing use of electronic health records (EHRs), the quality of EHRs is problematic. Efforts have been made to address reasons for poor EHR documentation quality. Previous systematic reviews have assessed intervention effectiveness within the outpatient setting or paper documentation. The purpose of this systematic review was to assess the effectiveness of interventions seeking to improve EHR documentation within an inpatient setting.

MATERIALS AND METHODS

A search strategy was developed based on elaborated inclusion/exclusion criteria. Four databases, gray literature, and reference lists were searched. A REDCap data capture form was used for data extraction, and study quality was assessed using a customized tool. Data were analyzed and synthesized in a narrative, semiquantitative manner.

RESULTS

Twenty-four studies were included in this systematic review. Owing to high heterogeneity, quantitative comparison was not possible. However, statistically significant results in interventions and affected outcomes were analyzed and discussed. Education and implementation of a new EHR reporting system were the most successful interventions, as evidenced by significantly improved EHR documentation.

DISCUSSION

Heterogeneity of interventions, outcomes, document type, EHR user, and other variables led to difficulty in measuring EHR documentation quality and effectiveness of interventions. However, the use of education as a primary intervention aligned closely with existing literature in similar fields.

CONCLUSIONS

Interventions implemented to enhance EHR documentation are highly variable and require standardization. Emphasis should be placed on this novel area of research to improve communication between healthcare providers and facilitate data sharing between centers and countries.

PROSPERO Registration Number: CRD42017083494.

Resident Workshop to Improve Inpatient Documentation Using the Progress Note Assessment and Plan Evaluation (PNAPE) Tool

INTRODUCTION

Physicians enter residency with varied knowledge regarding the purpose of progress notes and proficiency writing them. The objective of this study was to test whether resident knowledge, beliefs, and confidence writing inpatient progress notes improved after a 2.5-hour workshop intervention.

METHODS

An educational workshop and note assessment tool was constructed by resident and faculty stakeholders based on a review of literature and institutional best practices. The Progress Note Assessment and Plan Evaluation (PNAPE) tool was designed to assess adherence to best practices in the assessment and plan section of progress notes. Thirty-four residents from a midsized pediatric residency program attended the workshop, which consisted of didactics and small-group work evaluating sample notes using the PNAPE tool. Participants completed a four-question online pre- and postworkshop survey to evaluate their knowledge of progress note components and attitudes regarding note importance. Pre-post analysis was performed with Chi-square testing for true/false questions, and Mann-Whitney testing for Likert scale questions and summative scores.

RESULTS

A majority of pediatric residents completed the preintervention (n = 26, 76% response rate) and postintervention (n = 23, 68% response rate) surveys. Accurate response rate improved in 15 of 20 of the true/false items, with a statistically significant improvement in five items. Resident perceptions of note importance and confidence in note writing also increased.

DISCUSSION

A workshop intervention may effectively educate pediatric residents about progress note best practices. Further studies should assess the impact of the intervention on sustained knowledge and beliefs about progress notes and subsequent note quality.

Improved efficiency of patient admission with electronic health records in neurosurgery

BACKGROUND

Electronic health records (EHRs) may be controversial but they have the potential to improve patient care. We investigated whether the introduction of an electronic template-based admission form for the collection of information about the patient's medical history and neurological and clinical state at admission in the neurosurgical unit might have an impact on the quality of documentation in a discharge record and the amount of time taken to produce this documentation.

METHOD

A new digital template-based admission form (EHR) was developed and assessed with QNOTE, an assessment tool of medical notes with standardised criteria and the possibility to benchmark the quality of documentations. This was compared to 30 prior paper-based handwritten documentations (HWD) regarding the utilisation of these medical notes for dictation of medical discharge records.

RESULTS

Implementation of the EHR significantly improved the quality of patient admission documentation with a QNOTE mean grand score of 87 ± 22 (p < 0.0001) compared to prior HWD with 44 ± 30. The mean documentation time for HWD was 8.1 min ± 4.1 min and the dictation time for discharge records was 10.6 min ± 3.5 min. After implementation of EHR, the documentation time increased slightly to 9.6 min ± 2.3 min (n.s.), while the time for dictation of discharge records was reduced to 5.1 min ± 1.2 min (p < 0.0001). There was a clear correlation between a higher quality of documentation and a higher needed documentation time as well as higher quality of documentation and lower dictation times of discharge records.

CONCLUSION

Implementation of the EHR improved the quality of patient admission documentation and reduced the dictation time of discharge records.

IMPLICATIONS

It is crucial to involve stakeholders and users of EHRs in a timely manner during the stage of development and implementation phase to ensure optimal results and better usability.

Development and Establishment of Initial Validity Evidence for a Novel Tool for Assessing Trainee Admission Notes

BACKGROUND

Documentation is a key component of practice, yet few curricula have been published to teach trainees proper note construction. Additionally, a gold standard for assessing note quality does not exist, and no documentation assessment tools integrate with established competency-based frameworks.

OBJECTIVE

To develop and establish initial validity evidence for a novel tool that assesses key components of trainee admission notes and maps to the Accreditation Council for Graduate Medical Education (ACGME) milestone framework.

DESIGN

Using an iterative, consensus building process we developed the Admission Note Assessment Tool (ANAT). Pilot testing was performed with both the supervising attending and study team raters not involved in care of the patients. The finalized tool was piloted with attendings from other institutions.

PARTICIPANTS

Local experts participated in tool development and pilot testing. Additional attending physicians participated in pilot testing.

MAIN MEASURES

Content, response process, and internal structure validity evidence was gathered using Messick's framework. Inter-rater reliability was assessed using percent agreement.

KEY RESULTS

The final tool consists of 16 checklist items and two global assessment items. Pilot testing demonstrated rater agreement of 72% to 100% for checklist items and 63% to 70% for global assessment items. Note assessment required an average of 12.3 min (SD 3.7). The study generated validity evidence in the domains of content, response process, and internal structure for use of the tool in rating admission notes.

CONCLUSIONS

The ANAT assesses individual components of a note, incorporates billing criteria, targets note "bloat," allows for narrative feedback, and provides global assessments mapped to the ACGME milestone framework. The ANAT can be used to assess admission notes by any attending and at any time after note completion with minimal rater training. The ANAT allows programs to implement routine note assessment for multiple functions with the use of a single tool.

Pilot trial of semi-automated medical note writing using lexeme hypotheses

Clinicians write a billion free text notes per year. These notes are typically replete with errors of all types. No established automated method can extract data from this treasure trove. The practice of medicine therefore remains haphazard and chaotic, resulting in vast economic waste. The lexeme hypotheses are based on our analysis of how records are created. They enable a computer system to predict what issue a clinician will need to address next, based on the environment in which the clinician is working, and what responses the clinician has selected to date. The system uses a lexicon storing the issues (queries) and a range of responses to the issues. When the clinician selects a response, a text fragment is added to the output file. In the first phase of this work, the notes of 69 returning hemophilia patients were scrutinized, and the lexicon was expanded to 847 lexeme queries and 7995 responses to enable the construction of completed notes. The quality of lexeme-generated notes from 20 consecutive subjects was then compared to the clinicians' conventional clinic notes. The system generated grammatically correct notes. In comparison to the traditional clinic note, the lexeme-generated notes were more complete (88 % compared with 62 %), and had less typographical and grammatical errors (0.8 versus 3.5 errors per note). The system notes and traditional notes averaged about 800 words, but the traditional notes had a much wider distribution of lengths. The note-creation rate from marshalling the data to completion using the system averaged 80 wpm, twice as fast as the typical clinician can type. The lexeme method generates more complete, grammatical and organized notes faster than traditional methods. The notes are completely computerized at inception, and they incorporate prompts for clinicians to address otherwise overlooked items. This pilot justifies further exploration of this methodology.

Cross-Cover Documentation: Multicenter Development of Assessment Tool for Quality Improvement

Construct: We aimed to develop an assessment tool to measure the quality of electronic health record inpatient documentation of cross-cover events. Background: Cross-cover events occur in hospitalized patients when the primary team is absent. Documentation is critical for safe transitions of care. The quality of documentation for cross-cover events remains unknown, and no standardized tool exists for assessment. Approach: We created an assessment tool for cross-cover note quality with content validation based on input from 15 experts. We measured interrater reliability of the tool and scored cross-cover note quality for hospitalized patients with overnight rapid response team activation on internal medicine services at 2 academic hospitals for 1 year. Patients with a code blue or a clinically insignificant event were excluded. The presence of a note, writer identity (resident or faculty), time from rapid response to documentation, note content (subjective and objective information, diagnosis, and plan), and patient outcomes were compared. Results: The instrument included 8 items to determine quality of cross-cover documentation: reason for physician notification, note written within 6 hours, subjective and objective patient information, diagnosis, treatment, level of care, and whether the attending physician was notified. The mean Cohen's kappa coefficient demonstrated good interrater agreement at 0.76. The instrument was scored in 222 patients with cross-cover notes. Notes documented by faculty scored higher in quality than residents (89% vs. 74% of 8 items present, p < .001). Cross-cover notes often lacked subjective information, diagnosis, and notification of attending, which was present in 60%, 62%, and 7% of notes, respectively. Conclusions: This study presents reliability evidence for an 8-item assessment tool to measure quality of documentation of cross-cover events and indicates improvement is needed for cross-cover education and safe transitions of care in acutely decompensating medical patients.

Evaluation of interventions to improve electronic health record documentation within the inpatient setting: a protocol for a systematic review

BACKGROUND

Electronic health records (EHRs) are increasing in popularity across national and international healthcare systems. Despite their augmented availability and use, the quality of electronic health records is problematic. There are various reasons for poor documentation quality within the EHR, and efforts have been made to address these areas. Previous systematic reviews have assessed intervention effectiveness within the outpatient setting or within paper documentation. This systematic review aims to assess the effectiveness of different interventions seeking to improve EHR documentation within an inpatient setting.

METHODS

We will employ a comprehensive search strategy that encompasses four distinct themes: EHR, documentation, interventions, and study design. Four databases (MEDLINE, EMBASE, CENTRAL, and CINAHL) will be searched along with an in-depth examination of the grey literature and reference lists of relevant articles. A customized hybrid study quality assessment tool has been designed, integrating components of the Downs and Black and Newcastle-Ottawa Scales, into a REDCap data capture form to facilitate data extraction and analysis. Given the predicted high heterogeneity between studies, it may not be possible to standardize data for a quantitative comparison and meta-analysis. Thus, data will be synthesized in a narrative, semi-quantitative manner.

DISCUSSION

This review will summarize the current level of evidence on the effectiveness of interventions implemented to improve inpatient EHR documentation, which could ultimately enhance data quality in administrative health databases.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42017083494.

Bibliometric analysis of the International Medical Informatics Association official journals

Objectives: This research article aims to analyze the bibliometric characteristics of four official International Medical Informatics Association (IMIA) journals, namely: the International Journal of Medical Informatics, Methods of Information in Medicine, Applied Clinical Informatics, and Informatics for Health and Social Care.Method: We used descriptive bibliometrics to study the trends of literature production, identify documents` types, most prolific authors, institutions, countries, and most cited publications of all four IMIA journals. Additionally, we visualized the content of published publications using bibliometric mapping to identify journals' main themes and the most prolific and most cited research terms.Results: In total, 6,837 publications were published in all four IMIA journals. Among them, there were 5,137 original articles, meaning that articles were the leading document type. Research is being conducted globally among various research institutions. The most prolific countries are the United States of America, the United Kingdom, Germany, the Netherlands, and Canada. Thematic analyses of clusters show that themes are overlapping between all four journals.Conclusion: The journals contribute to the advances in technology related to health information systems, knowledge-based and decision-making systems, health literacy, and electronic health records.

Asynchronous Speech Recognition Affects Physician Editing of Notes

OBJECTIVE

Clinician progress notes are an important record for care and communication, but there is a perception that electronic notes take too long to write and may not accurately reflect the patient encounter, threatening quality of care. Automatic speech recognition (ASR) has the potential to improve clinical documentation process; however, ASR inaccuracy and editing time are barriers to wider use. We hypothesized that automatic text processing technologies could decrease editing time and improve note quality. To inform the development of these technologies, we studied how physicians create clinical notes using ASR and analyzed note content that is revised or added during asynchronous editing.

MATERIALS AND METHODS

We analyzed a corpus of 649 dictated clinical notes from 9 physicians. Notes were dictated during rounds to portable devices, automatically transcribed, and edited later at the physician's convenience. Comparing ASR transcripts and the final edited notes, we identified the word sequences edited by physicians and categorized the edits by length and content.

RESULTS

We found that 40% of the words in the final notes were added by physicians while editing: 6% corresponded to short edits associated with error correction and format changes, and 34% were associated with longer edits. Short error correction edits that affect note accuracy are estimated to be less than 3% of the words in the dictated notes. Longer edits primarily involved insertion of material associated with clinical data or assessment and plans. The longer edits improve note completeness; some could be handled with verbalized commands in dictation.

CONCLUSION

Process interventions to reduce ASR documentation burden, whether related to technology or the dictation/editing workflow, should apply a portfolio of solutions to address all categories of required edits. Improved processes could reduce an important barrier to broader use of ASR by clinicians and improve note quality.

Using voice to create inpatient progress notes: effects on note timeliness, quality, and physician satisfaction

Objectives

We describe the evaluation of a system to create hospital progress notes using voice and electronic health record integration to determine if note timeliness, quality, and physician satisfaction are improved.

Materials and methods

We conducted a randomized controlled trial to measure effects of this new method of writing inpatient progress notes, which evolved over time, on important outcomes.

Results

Intervention subjects created 709 notes and control subjects created 1143 notes. When adjusting for clustering by provider and secular trends, there was no significant difference between the intervention and control groups in the time between when patients were seen on rounds and when progress notes were viewable by others (95% confidence interval -106.9 to 12.2 min). There were no significant differences in physician satisfaction or note quality between intervention and control.

Discussion

Though we did not find support for the superiority of this system (Voice-Generated Enhanced Electronic Note System [VGEENS]) for our 3 primary outcomes, if notes are created using voice during or soon after rounds they are available within 10 min. Shortcomings that likely influenced subject satisfaction include the early state of our VGEENS and the short interval for system development before the randomized trial began.

Conclusion

VGEENS permits voice dictation on rounds to create progress notes and can reduce delay in note availability and may reduce dependence on copy/paste within notes. Timing of dictation determines when notes are available. Capturing notes in near-real-time has potential to apply NLP and decision support sooner than when notes are typed later in the day, and to improve note accuracy.

A Prescription for Note Bloat: An Effective Progress Note Template

BACKGROUND

United States hospitals have widely adopted electronic health records (EHRs). Despite the potential for EHRs to increase efficiency, there is concern that documentation quality has suffered.

OBJECTIVE

To examine the impact of an educational session bundled with a progress note template on note quality, length, and timeliness.

DESIGN

A multicenter, nonrandomized prospective trial.

SETTING

Four academic hospitals across the United States.

PARTICIPANTS

Intern physicians on inpatient internal medicine rotations at participating hospitals.

INTERVENTION

A task force delivered a lecture on current issues with documentation and suggested that interns use a newly designed best practice progress note template when writing daily progress notes.

MEASUREMENTS

Note quality was rated using a tool designed by the task force comprising a general impression score, the validated Physician Documentation Quality Instrument, 9-item version (PDQI-9), and a competency questionnaire. Reviewers documented number of lines per note and time signed.

RESULTS

Two hundred preintervention and 199 postintervention notes were collected. Seventy percent of postintervention notes used the template. Significant improvements were seen in the general impression score, all domains of the PDQI-9, and multiple competency items, including documentation of only relevant data, discussion of a discharge plan, and being concise while adequately complete. Notes had approximately 25% fewer lines and were signed on average 1.3 hours earlier in the day.

CONCLUSIONS

The bundled intervention for progress notes significantly improved the quality, decreased the length, and resulted in earlier note completion across 4 academic medical centers.

Electronic health record reviews to measure diagnostic uncertainty in primary care

RATIONALE, AIMS AND OBJECTIVES

Diagnostic uncertainty is common in primary care. Because it is challenging to measure, there is inadequate scientific understanding of diagnostic decision-making during uncertainty. Our objective was to understand how diagnostic uncertainty was documented in the electronic health record (EHR) and explore a strategy to retrospectively identify it using clinician documentation.

METHODS

We reviewed the literature to identify documentation language that could identify both direct expression and indirect inference of diagnostic uncertainty and designed an instrument to facilitate record review. Direct expression included clinician's use of question marks, differential diagnoses, symptoms as diagnosis, or vocabulary such as "probably, maybe, likely, unclear or unknown," while describing the diagnosis. Indirect inference included absence of documented diagnosis at the end of the visit, ordering of multiple consultations or diagnostic tests to resolve diagnostic uncertainty, and use of suspended judgement, test of treatment, and risk-averse disposition. Two physician-reviewers independently reviewed notes on a sample of outpatient visits to identify diagnostic uncertainty at the end of the visit. Documented Ninth Revision of the International Classification of Diseases (ICD-9) diagnosis codes and note quality were assessed.

RESULTS

Of 389 patient records reviewed, 218 had evidence of diagnostic activity and were included. In 156 visits (71.6%), reviewers identified clinicians who experienced diagnostic uncertainty with moderate inter-reviewer agreement (81.7%; Cohen's kappa: 0.609). Most cases (125, 80.1%) showed evidence of both direct expression and indirect inference. Uncertainty was directly expressed in 139 (89.1%) cases, most commonly by using symptoms as diagnosis (98, 62.8%), and inferred in 144 (92.3%). In more than 1/3 of visits (58, 37.2%), diagnostic uncertainty was recorded inappropriately using ICD-9 codes.

CONCLUSIONS

While current diagnosis coding mechanisms (ICD-9 and ICD-10) are unable to capture uncertainty, our study finds that review of EHR documentation can help identify diagnostic uncertainty with moderate reliability. Better measurement and understanding of diagnostic uncertainty could help inform strategies to improve the safety and efficiency of diagnosis.

What are they trying to do?: An analysis of Action Identities in using electronic documentation in an EHR

Documentation processes have changed substantially with EHR adoption. User satisfaction studies have focused on usability or cognitive analysis perspectives. Few studies have provided useful information to developers to improve designs. The purpose of this study is to report a 3-pronged approach to deepen understanding of the documentation process, with the intent to provide useful information for future design. This study was conducted in two phases, beginning with cognitive task interviews and observations, followed by post-observation interviews. Twenty-five constructs were identified across the phases, and we observed several patterns of note writing. Participants provided useful information to potentially inform future design. Our study illustrates how electronic documentation serves many clinical processes and is at the core of the medical record. Providers need multiple kinds of notes and ways to display notes. In order to meet provider goals, we must completely re-think the way electronic documentation is composed and displayed.

Use of dictation as a tool to decrease documentation errors in electronic health records

Background: Use of Electronic Health Records is increasing. Copy-and-paste function is frequently used with higher rates of documentation errors. Studies to determine the nature of such errors are needed.Objectives: Determination of the effect of implementing a dictation system for completing notes on the quality of clinical documentation. We hypothesized that implementation of the dictation system for note writing would decrease the rate of errors in the progress notes as well as decrease the rate of copying and pasting. Design/Methods: A prospective interventional study in inpatient medical service for six months’ duration starting in July 2016. Resident physicians’ charts were reviewed by the attending physician on a daily basis. This study was done in a community based hospital affiliated to a university program. Residents’ physicians included Internal Medicine, Transitional year and Combined Internal Medicine Pediatrics residents. Charts reviewed for hospitalized patients. A total of 54 residents were offered a pre-intervention survey indicating their subjective use of copy/paste function. Response rate of 85.18%. Progress notes were reviewed on a daily basis for residents on their inpatient rotation. A total of 621 notes were reviewed. Results: Percentage of notes copied prior to the intervention was 92.73% which decreased to 49.71% post-intervention (RR of 0.54, 95% CI 0.48 0.60 Z statistic 11.005 with p-value &lt;0.0001). Of the copied notes percentage of errors pre-intervention was 58% with no errors identified post-intervention (RR of 0.005, 95% CI 0.0003 0.0795 Z statistic 3.752 with p-value 0.0002). Most of the errors are from notes copied by the same author (85.8%). The most common documentation error was in the physical examination section. Conclusion: Implementing a dictation system eliminated documentation errors over our six months’ study. Further studies are needed to check long effects of using such systems on documentation errors

Promoting Responsible Electronic Documentation: Validity Evidence for a Checklist to Assess Progress Notes in the Electronic Health Record

Construct: We aimed to develop an instrument to measure the quality of inpatient electronic health record- (EHR-) generated progress notes without requiring raters to review the detailed chart or know the patient.

BACKGROUND

Notes written in EHRs have generated criticism for being unnecessarily long and redundant, perpetuating inaccuracy and obscuring providers' clinical reasoning. Available assessment tools either focus on outpatient progress notes or require chart review by raters to develop familiarity with the patient.

APPROACH

We used medical literature, local expert review, and attending focus groups to develop and refine an instrument to evaluate inpatient progress notes. We measured interrater reliability and scored the selected-response elements of the checklist for a sample of 100 progress notes written by PGY-1 trainees on the general medicine service.

RESULTS

We developed an instrument with 18 selected-response items and four open-ended items to measure the quality of inpatient progress notes written in the EHR. The mean Cohen's kappa coefficient demonstrated good agreement at .67. The mean note score was 66.9% of maximum possible points (SD = 10.6, range = 34.4%-93.3%).

CONCLUSIONS

We present validity evidence in the domains of content, internal structure, and response process for a new checklist for rating inpatient progress notes. The scored checklist can be completed in approximately 7 minutes by a rater who is not familiar with the patient and can be done without extensive chart review. We further demonstrate that trainee notes show substantial room for improvement.

The 9-Item Physician Documentation Quality Instrument (PDQI-9) score is not useful in evaluating EMR (scribe) note quality in Emergency Medicine

BACKGROUND

Scribes are assisting Emergency Physicians by writing their electronic clinical notes at the bedside during consultations. They increase physician productivity and improve their working conditions. The quality of Emergency scribe notes is unevaluated and important to determine.

OBJECTIVE

The primary objective of the study was to determine if the quality of Emergency Department scribe notes was equivalent to physician only notes, using the Physician Documentation Quality Instrument, Nine-item tool (PDQI-9).

METHODS

This was a retrospective, observational study comparing 110 scribed to 110 non-scribed Emergency Physician notes written at Cabrini Emergency Department, Australia. Consultations during a randomised controlled trial of scribe/doctor productivity in 2016 were used. Emergency physicians and nurses rated randomly selected, blinded and de-identified notes, 2 raters per note. Comparisons were made between paired scribed and unscribed notes and between raters of each note. Characteristics of individual raters were examined. The ability of the tool to discriminate between good and poor notes was tested.

RESULTS

The PDQI-9 tool has significant issues. Individual items had good internal consistency (Cronbach's alpha=0.93), but there was very poor agreement between raters (Pearson's r=0.07, p=0.270). There were substantial differences in PDQI-9 scores allocated by each rater, with some giving typically lower scores than others, F(25,206)=1.93, p=0.007. The tool was unable to distinguish good from poor notes, F(3,34)=1.15, p=0.342. There was no difference in PDQI-9 score between scribed and non-scribed notes.

CONCLUSIONS

The PDQI-9 documentation quality tool did not demonstrate reliability or validity in evaluating Emergency Medicine consultation notes. We found no evidence that scribed notes were of poorer quality than non-scribed notes, however Emergency scribe note quality has not yet been determined.