1
|
Schaye V, Guzman B, Burk-Rafel J, Marin M, Reinstein I, Kudlowitz D, Miller L, Chun J, Aphinyanaphongs Y. Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning Documentation. J Gen Intern Med 2022; 37:2230-2238. [PMID: 35710676 PMCID: PMC9296753 DOI: 10.1007/s11606-022-07526-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 03/29/2022] [Indexed: 11/29/2022]
Abstract
BACKGROUND Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment. OBJECTIVE The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes. DESIGN, PARTICIPANTS, MAIN MEASURES Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend. KEY RESULTS The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications). CONCLUSIONS The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.
Collapse
Affiliation(s)
- Verity Schaye
- NYU Grossman School of Medicine, New York, NY, USA. .,NYC Health & Hospitals/Bellevue, New York, NY, USA.
| | | | | | - Marina Marin
- NYU Grossman School of Medicine, New York, NY, USA
| | | | | | - Louis Miller
- Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| | - Jonathan Chun
- Stanford University School of Medicine, Stanford, CA, USA
| | | |
Collapse
|
2
|
Kim M, Chan N, Evans J, Min JK, Hayton AC. Improving Medical Student Inpatient Documentation Through Feedback Using a Note Assessment Tool. Cureus 2022; 14:e23369. [PMID: 35475068 PMCID: PMC9020806 DOI: 10.7759/cureus.23369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/11/2022] [Indexed: 11/05/2022] Open
Abstract
Introduction Documentation within the Electronic Health Record (EHR) is an essential skill for medical students to succeed in residency and post-residency training. The increased use of medical student progress notes for billable services raises the need for the education and assessment of quality note writing. We hypothesized that structured note feedback using a note assessment tool would improve the quality of medical student inpatient progress notes. Methods We conducted a retrospective study to review the quality of student inpatient progress notes written before and after structured feedback using the Responsible Electronic Documentation (RED) checklist throughout a third-year internal medicine clerkship. The first intervention group received feedback from clerkship directors in the 2017-2018 academic year and the second intervention group received feedback from ward residents/attendings in the 2018-2019 academic year. Within each intervention group, the total note scores from pre and post-intervention were compared. Results Feedback from clerkship directors yielded a greater increase in students’ total note score from pre to post-intervention compared to ward resident/attending feedback (F(1,255) = 12.84, p < 0.001). Cohen’s d effect size value was greater for the clerkship director feedback arm (d=0.71) compared to the ward resident/attending feedback arm (d=0.24). Post-hoc analyses using dependent sample t-tests revealed that there were significant increases in total note scores from pre to post-intervention for both the clerkship director arm (t(123) = 8.26, p < 0.001, d = 0.71) and the ward resident/attending arm (t(132) = 2.85, p = 0.005, d = 0.24). Conclusion Clerkship director feedback led to a greater increase in medical student documentation compared to ward attending/resident feedback. Nonetheless, structured feedback with a note assessment tool, whether from clerkship directors or ward attendings/residents, leads to a significant improvement in medical student documentation. Though there are various methods for providing feedback, educators can use the RED checklist to provide clear guidelines that will facilitate note-writing feedback.
Collapse
|
3
|
Schaye V, Miller L, Kudlowitz D, Chun J, Burk-Rafel J, Cocks P, Guzman B, Aphinyanaphongs Y, Marin M. Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback. J Gen Intern Med 2022; 37:507-512. [PMID: 33945113 PMCID: PMC8858363 DOI: 10.1007/s11606-021-06805-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/10/2020] [Accepted: 04/03/2021] [Indexed: 02/03/2023]
Abstract
BACKGROUND Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability. OBJECTIVE Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool. DESIGN, PARTICIPANTS, AND MAIN MEASURES The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting. KEY RESULTS The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality. CONCLUSIONS The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.
Collapse
Affiliation(s)
- Verity Schaye
- NYU Grossman School of Medicine, New York, NY, USA. .,NYC Health + Hospitals/Bellevue, New York, NY, USA.
| | - Louis Miller
- Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| | | | - Jonathan Chun
- Stanford University School of Medicine, Stanford, CA, USA
| | | | | | | | | | - Marina Marin
- NYU Grossman School of Medicine, New York, NY, USA
| |
Collapse
|
4
|
Bacon CEW, Kasamatsu TM, Lam KC, Nottingham SL. Future Strategies to Enhance Patient Care Documentation Among Athletic Trainers: A Report From the Athletic Training Practice-Based Research Network. J Athl Train 2018; 53:619-626. [PMID: 29893602 DOI: 10.4085/1062-6050-298-17] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
CONTEXT High-quality patient care documentation is an essential component of any health care professional's daily practice. Whereas athletic trainers (ATs) recognize the importance of patient care documentation, several barriers may prevent them from producing high-quality patient care documentation. OBJECTIVE To explore beneficial strategies and techniques that ATs perceived would enhance the quality of patient care documentation in the secondary school setting. DESIGN Qualitative study. SETTING Individual telephone interviews. PATIENTS OR OTHER PARTICIPANTS Ten ATs who were members of the Athletic Training Practice-Based Research Network and employed in the secondary school setting were interviewed (4 men, 6 women with 7.1 ± 7.8 years of athletic training experience). DATA COLLECTION AND ANALYSIS An individual telephone interview was conducted with each participant. Once transcribed, data were analyzed into common themes and categories per the consensual qualitative research tradition. Trustworthiness of the data was achieved through triangulation strategies: (1) the inclusion of multiple researchers to ensure accuracy and representativeness of the data and (2) participant member checking. RESULTS Participants identified several documentation strategies they perceived would be helpful to improve the quality of patient care documentation, including mode and consistency of documentation and the need for a standardized process as well as the need for system standardization. In addition, participants discussed the need for more education on patient care documentation. Specifically, they identified ways of learning and strategies for future education to enhance patient care documentation across the profession. CONCLUSIONS As athletic training continues to evolve, it is crucial that ATs are well educated on how to produce high-quality patient care documentation as a part of routine practice. Continuing professional development opportunities are needed to promote lifelong learning in the area of patient care documentation.
Collapse
|
5
|
DeLeon S, Mothner B, Middleman A. Improving student documentation using a feedback tool. CLINICAL TEACHER 2017; 15:48-51. [PMID: 28225204 DOI: 10.1111/tct.12625] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
BACKGROUND Medical students are often given non-standardised midpoint feedback and final evaluation on their history and physical examination documentation, despite this written communication being one of the most important aspects of patient care. This study aims to assess the effect of using a standardised feedback tool on overall student documentation performance. METHODS A standardised written evaluation form, called the history and physical examination write-up assessment (HAPA) form, was previously developed and published in The Clinical Teacher in 2011. This form evaluates the documentation of student communication, data collection and clinical reasoning, and was used to provide midpoint documentation feedback to one group of medical students, whereas a second group received non-standardised midpoint feedback on documentation. Final written history and physical examination documentation from both groups was then evaluated using the standardised tool to assess the effect of receiving more formalised midpoint documentation feedback using the HAPA form. RESULTS The group receiving standardised midpoint feedback using the HAPA form performed significantly better on the final evaluations of their documentation, especially when examining communication and data collection documentation. Performance in clinical reasoning documentation did not demonstrate a significant change between the two groups. Medical students are often given non-standardised midpoint feedback and final evaluation CONCLUSIONS: The use of a standardised evaluation tool, such as the HAPA form, may be associated with improved student documentation. This form provides educators with an effective tool to use when providing feedback and final evaluation of medical student documentation.
Collapse
Affiliation(s)
- Stephanie DeLeon
- Pediatrics Department, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma, USA
| | - Brent Mothner
- Pediatrics Department, Baylor College of Medicine, Houston, Texas, USA
| | - Amy Middleman
- Pediatrics Department, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma, USA
| |
Collapse
|
6
|
King MA, Phillipi CA, Buchanan PM, Lewin LO. Developing Validity Evidence for the Written Pediatric History and Physical Exam Evaluation Rubric. Acad Pediatr 2017; 17:68-73. [PMID: 27521461 DOI: 10.1016/j.acap.2016.08.001] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/17/2016] [Revised: 07/29/2016] [Accepted: 08/03/2016] [Indexed: 11/28/2022]
Abstract
OBJECTIVE The written history and physical examination (H&P) is an underutilized source of medical trainee assessment. The authors describe development and validity evidence for the Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric: a novel tool for evaluating written H&Ps. METHODS Using an iterative process, the authors drafted, revised, and implemented the 10-item rubric at 3 academic institutions in 2014. Eighteen attending physicians and 5 senior residents each scored 10 third-year medical student H&Ps. Inter-rater reliability (IRR) was determined using intraclass correlation coefficients. Cronbach α was used to report consistency and Spearman rank-order correlations to determine relationships between rubric items. Raters provided a global assessment, recorded time to review and score each H&P, and completed a rubric utility survey. RESULTS Overall intraclass correlation was 0.85, indicating adequate IRR. Global assessment IRR was 0.89. IRR for low- and high-quality H&Ps was significantly greater than for medium-quality ones but did not differ on the basis of rater category (attending physician vs. senior resident), note format (electronic health record vs nonelectronic), or student diagnostic accuracy. Cronbach α was 0.93. The highest correlation between an individual item and total score was for assessments was 0.84; the highest interitem correlation was between assessment and differential diagnosis (0.78). Mean time to review and score an H&P was 16.3 minutes; residents took significantly longer than attending physicians. All raters described rubric utility as "good" or "very good" and endorsed continued use. CONCLUSIONS The P-HAPEE rubric offers a novel, practical, reliable, and valid method for supervising physicians to assess pediatric written H&Ps.
Collapse
Affiliation(s)
- Marta A King
- Division of General Academic Pediatrics, Saint Louis University School of Medicine, St Louis, Mo.
| | - Carrie A Phillipi
- Department of Pediatrics, Oregon Health & Science University, Portland, OR
| | - Paula M Buchanan
- Center for Outcomes Research, Saint Louis University, St Louis, Mo
| | - Linda O Lewin
- Department of Pediatrics, University of Maryland School of Medicine, Bethesda, MD
| |
Collapse
|
7
|
Bynum D, Colford C, Royal K. Teaching medical students the art of the 'write-up'. CLINICAL TEACHER 2015; 12:246-9. [PMID: 26036763 DOI: 10.1111/tct.12304] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
BACKGROUND The creation of a complete 'write-up' continues to be essential to the clinical learning experience for medical students. The ability to document a clinical encounter is a key communication skill and Core Entrustable Professional Activity for entering residency. METHODS We developed a guide to the comprehensive write-up, a grading rubric, and a videotaped encounter with a standardised doctor and patient. Second-year medical students created a write-up based upon this encounter, which was then peer-reviewed in a small group writer's workshop session. The students were later required to submit a write-up, based upon a real patient encounter, to the course directors for a grade. All write-ups (n = 185) were graded by the course director. Fifty-one were independently graded by a second course director. These grades were compared with the 175 student write-ups from the previous year. The ability to document a clinical encounter is a key communication skill … for entering residency RESULTS The average grade for student write-ups was 86 with a standard deviation of 9, compared with an average of 75 with a standard deviation of 17 for the year prior to the introduction of this session (p < 0.001). The average score given by a second rater was 83 with a standard deviation of 11, indicating a high level of agreement and internal consistency. DISCUSSION These tools were easy to use and well received by faculty members and students, and the quality of student write-ups significantly improved after the introduction of the session. The grading rubric demonstrated high inter-rater reliability, indicating that this can be adapted and used by others for instruction and assessment.
Collapse
Affiliation(s)
- Debra Bynum
- Department of Internal Medicine, University of North Carolina, North Carolina, USA
| | - Cristin Colford
- Department of Internal Medicine, University of North Carolina, North Carolina, USA
| | - Kenneth Royal
- Department of Family Medicine, University of North Carolina, North Carolina, USA
| |
Collapse
|