Reliability of the history and physical assessment (HAPA) form

Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning Documentation

BACKGROUND

Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment.

OBJECTIVE

The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes.

DESIGN, PARTICIPANTS, MAIN MEASURES

Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend.

KEY RESULTS

The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications).

CONCLUSIONS

The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.

Improving Medical Student Inpatient Documentation Through Feedback Using a Note Assessment Tool

Introduction

Documentation within the Electronic Health Record (EHR) is an essential skill for medical students to succeed in residency and post-residency training. The increased use of medical student progress notes for billable services raises the need for the education and assessment of quality note writing. We hypothesized that structured note feedback using a note assessment tool would improve the quality of medical student inpatient progress notes.

Methods

We conducted a retrospective study to review the quality of student inpatient progress notes written before and after structured feedback using the Responsible Electronic Documentation (RED) checklist throughout a third-year internal medicine clerkship. The first intervention group received feedback from clerkship directors in the 2017-2018 academic year and the second intervention group received feedback from ward residents/attendings in the 2018-2019 academic year. Within each intervention group, the total note scores from pre and post-intervention were compared.

Results

Feedback from clerkship directors yielded a greater increase in students’ total note score from pre to post-intervention compared to ward resident/attending feedback (F(1,255) = 12.84, p < 0.001). Cohen’s d effect size value was greater for the clerkship director feedback arm (d=0.71) compared to the ward resident/attending feedback arm (d=0.24). Post-hoc analyses using dependent sample t-tests revealed that there were significant increases in total note scores from pre to post-intervention for both the clerkship director arm (t(123) = 8.26, p < 0.001, d = 0.71) and the ward resident/attending arm (t(132) = 2.85, p = 0.005, d = 0.24).

Conclusion

Clerkship director feedback led to a greater increase in medical student documentation compared to ward attending/resident feedback. Nonetheless, structured feedback with a note assessment tool, whether from clerkship directors or ward attendings/residents, leads to a significant improvement in medical student documentation. Though there are various methods for providing feedback, educators can use the RED checklist to provide clear guidelines that will facilitate note-writing feedback.

Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback

BACKGROUND

Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability.

OBJECTIVE

Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool.

DESIGN, PARTICIPANTS, AND MAIN MEASURES

The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting.

KEY RESULTS

The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality.

CONCLUSIONS

The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.

Future Strategies to Enhance Patient Care Documentation Among Athletic Trainers: A Report From the Athletic Training Practice-Based Research Network

CONTEXT

  High-quality patient care documentation is an essential component of any health care professional's daily practice. Whereas athletic trainers (ATs) recognize the importance of patient care documentation, several barriers may prevent them from producing high-quality patient care documentation.

OBJECTIVE

  To explore beneficial strategies and techniques that ATs perceived would enhance the quality of patient care documentation in the secondary school setting.

DESIGN

  Qualitative study.

SETTING

  Individual telephone interviews.

PATIENTS OR OTHER PARTICIPANTS

  Ten ATs who were members of the Athletic Training Practice-Based Research Network and employed in the secondary school setting were interviewed (4 men, 6 women with 7.1 ± 7.8 years of athletic training experience).

DATA COLLECTION AND ANALYSIS

  An individual telephone interview was conducted with each participant. Once transcribed, data were analyzed into common themes and categories per the consensual qualitative research tradition. Trustworthiness of the data was achieved through triangulation strategies: (1) the inclusion of multiple researchers to ensure accuracy and representativeness of the data and (2) participant member checking.

RESULTS

  Participants identified several documentation strategies they perceived would be helpful to improve the quality of patient care documentation, including mode and consistency of documentation and the need for a standardized process as well as the need for system standardization. In addition, participants discussed the need for more education on patient care documentation. Specifically, they identified ways of learning and strategies for future education to enhance patient care documentation across the profession.

CONCLUSIONS

  As athletic training continues to evolve, it is crucial that ATs are well educated on how to produce high-quality patient care documentation as a part of routine practice. Continuing professional development opportunities are needed to promote lifelong learning in the area of patient care documentation.

Improving student documentation using a feedback tool

BACKGROUND

Medical students are often given non-standardised midpoint feedback and final evaluation on their history and physical examination documentation, despite this written communication being one of the most important aspects of patient care. This study aims to assess the effect of using a standardised feedback tool on overall student documentation performance.

METHODS

A standardised written evaluation form, called the history and physical examination write-up assessment (HAPA) form, was previously developed and published in The Clinical Teacher in 2011. This form evaluates the documentation of student communication, data collection and clinical reasoning, and was used to provide midpoint documentation feedback to one group of medical students, whereas a second group received non-standardised midpoint feedback on documentation. Final written history and physical examination documentation from both groups was then evaluated using the standardised tool to assess the effect of receiving more formalised midpoint documentation feedback using the HAPA form.

RESULTS

The group receiving standardised midpoint feedback using the HAPA form performed significantly better on the final evaluations of their documentation, especially when examining communication and data collection documentation. Performance in clinical reasoning documentation did not demonstrate a significant change between the two groups. Medical students are often given non-standardised midpoint feedback and final evaluation CONCLUSIONS: The use of a standardised evaluation tool, such as the HAPA form, may be associated with improved student documentation. This form provides educators with an effective tool to use when providing feedback and final evaluation of medical student documentation.

Developing Validity Evidence for the Written Pediatric History and Physical Exam Evaluation Rubric

OBJECTIVE

The written history and physical examination (H&P) is an underutilized source of medical trainee assessment. The authors describe development and validity evidence for the Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric: a novel tool for evaluating written H&Ps.

METHODS

Using an iterative process, the authors drafted, revised, and implemented the 10-item rubric at 3 academic institutions in 2014. Eighteen attending physicians and 5 senior residents each scored 10 third-year medical student H&Ps. Inter-rater reliability (IRR) was determined using intraclass correlation coefficients. Cronbach α was used to report consistency and Spearman rank-order correlations to determine relationships between rubric items. Raters provided a global assessment, recorded time to review and score each H&P, and completed a rubric utility survey.

RESULTS

Overall intraclass correlation was 0.85, indicating adequate IRR. Global assessment IRR was 0.89. IRR for low- and high-quality H&Ps was significantly greater than for medium-quality ones but did not differ on the basis of rater category (attending physician vs. senior resident), note format (electronic health record vs nonelectronic), or student diagnostic accuracy. Cronbach α was 0.93. The highest correlation between an individual item and total score was for assessments was 0.84; the highest interitem correlation was between assessment and differential diagnosis (0.78). Mean time to review and score an H&P was 16.3 minutes; residents took significantly longer than attending physicians. All raters described rubric utility as "good" or "very good" and endorsed continued use.

CONCLUSIONS

The P-HAPEE rubric offers a novel, practical, reliable, and valid method for supervising physicians to assess pediatric written H&Ps.

Teaching medical students the art of the 'write-up'

BACKGROUND

The creation of a complete 'write-up' continues to be essential to the clinical learning experience for medical students. The ability to document a clinical encounter is a key communication skill and Core Entrustable Professional Activity for entering residency.

METHODS

We developed a guide to the comprehensive write-up, a grading rubric, and a videotaped encounter with a standardised doctor and patient. Second-year medical students created a write-up based upon this encounter, which was then peer-reviewed in a small group writer's workshop session. The students were later required to submit a write-up, based upon a real patient encounter, to the course directors for a grade. All write-ups (n = 185) were graded by the course director. Fifty-one were independently graded by a second course director. These grades were compared with the 175 student write-ups from the previous year. The ability to document a clinical encounter is a key communication skill … for entering residency

RESULTS

The average grade for student write-ups was 86 with a standard deviation of 9, compared with an average of 75 with a standard deviation of 17 for the year prior to the introduction of this session (p < 0.001). The average score given by a second rater was 83 with a standard deviation of 11, indicating a high level of agreement and internal consistency.

DISCUSSION

These tools were easy to use and well received by faculty members and students, and the quality of student write-ups significantly improved after the introduction of the session. The grading rubric demonstrated high inter-rater reliability, indicating that this can be adapted and used by others for instruction and assessment.