The IDEA Assessment Tool: Assessing the Reporting, Diagnostic Reasoning, and Decision-Making Skills Demonstrated in Medical Students' Hospital Admission Notes

Teaching clinical reasoning: principles from the literature to help improve instruction from the classroom to the bedside

Clinical reasoning has been characterized as being an essential aspect of being a physician. Despite this, clinical reasoning has a variety of definitions and medical error, which is often attributed to clinical reasoning, has been reported to be a leading cause of death in the United States and abroad. Further, instructors struggle with teaching this essential ability which often does not play a significant role in the curriculum. In this article, we begin with defining clinical reasoning and then discuss four principles from the literature as well as a variety of techniques for teaching these principles to help ground an instructors' understanding in clinical reasoning. We also tackle contemporary challenges in teaching clinical reasoning such as the integration of artificial intelligence and strategies to help with transitions in instruction (e.g., from the classroom to the clinic or from medical school to residency/registrar training) and suggest next steps for research and innovation in clinical reasoning.

Evaluating Clinical Reasoning in Undergraduate Medical Education: The Value of a Virtual Oral Assessment

BACKGROUND

Oral assessments are essential components of board certification in numerous fields, as they provide insight into problem-solving capacity and clinical reasoning. The development of clinical reasoning often begins in undergraduate medical education and remains a challenge to assess.

OBJECTIVE

We developed a pilot oral assessment to evaluate medical student oral presentations and systematically assess clinical reasoning. This was incorporated into a previously existing cumulative assessment at the conclusion of the third year of medical school, with the intent to demonstrate feasibility and future reliability of this exam format.

METHODS

This pilot oral assessment was developed using content taught during third year clerkships. A modified Assessment of Reasoning Tool (ART) was used as the evaluation metric. It was conducted virtually to include faculty members from multiple disciplines and accommodate schedules and space limitations.

RESULTS

A total of 152 third year medical students completed the exam, with a total of 15 faculty examiners. 89% of medical students scored as complete in hypothesis directed history, 93% in problem representation, 86% in prioritized differential diagnoses, and 67% in effectively directing management. Most examiners felt an oral assessment is effective to determine a medical student's clinical reasoning ability.

CONCLUSIONS

Virtual oral assessments of clinical reasoning can be incorporated in undergraduate medical education to identify students struggling with components of clinical reasoning, while also allowing maximum flexibility for the clinician educator workforce as examiners. Longitudinal use of these exams would be valuable to track the development of clinical reasoning across the medical school curriculum.

A Patient-Centered Documentation Skills Curriculum for Preclerkship Medical Students in an Open Notes Era

Introduction

New legislation allows patients (with permitted exceptions) to read their clinical notes, leading to both benefits and ethical dilemmas. Medical students need a robust curriculum to learn documentation skills within this challenging context. We aimed to teach note-writing skills through a patient-centered lens with special consideration for the impact on patients and providers. We developed this session for first-year medical students within their foundational clinical skills course to place bias-free language at the forefront of how they learn to construct a medical note.

Methods

One hundred seventy-three first-year medical and dental students participated in this curriculum. They completed an asynchronous presession module first, followed by a 2-hour synchronous workshop including a didactic, student-led discussion and sample patient note exercise. Students were subsequently responsible throughout the year for constructing patient-centered notes, graded by faculty with a newly developed rubric and checklist of best practices.

Results

On postworkshop surveys, learners reported increased preparedness in their ability to document in a patient-centered manner (presession M = 2.2, midyear M = 3.9, p < .001), as rated on a 5-point Likert scale (1 = not prepared at all, 5 = very prepared), and also found this topic valuable to learn early in their training.

Discussion

This curriculum utilizes a multipart approach to prepare learners to employ clinical notes to communicate with patients and providers, with special attention to how patients and their care partners receive a note. Future directions include expanding the curriculum to higher levels of learning and validating the developed materials.

Development and Validation of a Formative Assessment Tool for Nephrology Fellows' Clinical Reasoning

BACKGROUND

Diagnostic errors are commonly driven by failures in clinical reasoning. Deficits in clinical reasoning are common among graduate medical learners, including nephrology fellows. We created and validated an instrument to assess clinical reasoning in a national cohort of nephrology fellows and established performance thresholds for remedial coaching.

METHODS

Experts in nephrology education and clinical reasoning remediation designed an instrument to measure clinical reasoning through a written patient encounter note from a web-based, simulated AKI consult. The instrument measured clinical reasoning in three domains: problem representation, differential diagnosis with justification, and diagnostic plan with justification. Inter-rater reliability was established in a pilot cohort ( n =7 raters) of first-year nephrology fellows using a two-way random effects agreement intraclass correlation coefficient model. The instrument was then administered to a larger cohort of first-year fellows to establish performance standards for coaching using the Hofstee method ( n =6 raters).

RESULTS

In the pilot cohort, there were 15 fellows from four training program, and in the study cohort, there were 61 fellows from 20 training programs. The intraclass correlation coefficients for problem representation, differential diagnosis, and diagnostic plan were 0.90, 0.70, and 0.50, respectively. Passing thresholds (% total points) in problem representation, differential diagnosis, and diagnostic plan were 59%, 57%, and 62%, respectively. Fifty-nine percent ( n =36) met the threshold for remedial coaching in at least one domain.

CONCLUSIONS

We provide validity evidence for a simulated AKI consult for formative assessment of clinical reasoning in nephrology fellows. Most fellows met criteria for coaching in at least one of three reasoning domains, demonstrating a need for learner assessment and instruction in clinical reasoning.

Using an experiential learning model to teach clinical reasoning theory and cognitive bias: an evaluation of a first-year medical student curriculum

BACKGROUND

Most medical students entering clerkships have limited understanding of clinical reasoning concepts. The value of teaching theories of clinical reasoning and cognitive biases to first-year medical students is unknown. This study aimed to evaluate the value of explicitly teaching clinical reasoning theory and cognitive bias to first-year medical students.

METHODS

Using Kolb's experiential learning model, we introduced dual process theory, script theory, and cognitive biases in teaching clinical reasoning to first-year medical students at an academic medical center in New York City between January and June 2020. Due to the COVID-19 pandemic, instruction was transitioned to a distance learning format in March 2020. The curriculum included a series of written clinical reasoning examinations with facilitated small group discussions. Written self-assessments prompted each student to reflect on the experience, draw conclusions about their clinical reasoning, and plan for future encounters involving clinical reasoning. We evaluated the value of the curriculum using mixed-methods to analyze faculty assessments, student self-assessment questionnaires, and an end-of-curriculum anonymous questionnaire eliciting student feedback.

RESULTS

Among 318 total examinations of 106 students, 254 (80%) had a complete problem representation, while 199 (63%) of problem representations were considered concise. The most common cognitive biases described by students in their clinical reasoning were anchoring bias, availability bias, and premature closure. Four major themes emerged as valuable outcomes of the CREs as identified by students: (1) synthesis of medical knowledge; (2) enhanced ability to generate differential diagnoses; (3) development of self-efficacy related to clinical reasoning; (4) raised awareness of personal cognitive biases.

CONCLUSIONS

We found that explicitly teaching clinical reasoning theory and cognitive biases using an experiential learning model provides first-year medical students with valuable opportunities for developing knowledge, skills, and self-efficacy related to clinical reasoning.

Automated Patient Note Grading: Examining Scoring Reliability and Feasibility

PURPOSE

Scoring postencounter patient notes (PNs) yields significant insights into student performance, but the resource intensity of scoring limits its use. Recent advances in natural language processing (NLP) and machine learning allow application of automated short answer grading (ASAG) for this task. This retrospective study evaluated psychometric characteristics and reliability of an ASAG system for PNs and factors contributing to implementation, including feasibility and case-specific phrase annotation required to tune the system for a new case.

METHOD

PNs from standardized patient (SP) cases within a graduation competency exam were used to train the ASAG system, applying a feed-forward neural networks algorithm for scoring. Using faculty phrase-level annotation, 10 PNs per case were required to tune the ASAG system. After tuning, ASAG item-level ratings for 20 notes were compared across ASAG-faculty (4 cases, 80 pairings) and ASAG-nonfaculty (2 cases, 40 pairings). Psychometric characteristics were examined using item analysis and Cronbach's alpha. Inter-rater reliability (IRR) was examined using kappa.

RESULTS

ASAG scores demonstrated sufficient variability in differentiating learner PN performance and high IRR between machine and human ratings. Across all items the ASAG-faculty scoring mean kappa was .83 (SE ± .02). The ASAG-nonfaculty pairings kappa was .83 (SE ± .02). The ASAG scoring demonstrated high item discrimination. Internal consistency reliability values at the case level ranged from a Cronbach's alpha of .65 to .77. Faculty time cost to train and supervise nonfaculty raters for 4 cases was approximately $1,856. Faculty cost to tune the ASAG system was approximately $928.

CONCLUSIONS

NLP-based automated scoring of PNs demonstrated a high degree of reliability and psychometric confidence for use as learner feedback. The small number of phrase-level annotations required to tune the system to a new case enhances feasibility. ASAG-enabled PN scoring has broad implications for improving feedback in case-based learning contexts in medical education.

Augmented Reality Integration in Manikin-Based Simulations: Bringing Basic Science to the Critical Care Bedside with Limited Augmented Reality Resources

Immersive simulation and augmented reality (AR) are powerful educational tools in high-risk medical professions. Basic science AR, such as anatomic holograms, are gaining popularity. Many educators want to adopt AR and integrate basic science review in high-risk clinical decision-making but cannot afford it. In this project, we designed three AR integrated manikin-based simulations (ARI-MBS) by combining critical care scenarios with commercially available AR programs. Using a single headset and limited equipment, we technically integrated AR into MBS in a way that both students and faculty found rewarding. We present our design, so that others may replicate it.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01821-z.

Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning Documentation

BACKGROUND

Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment.

OBJECTIVE

The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes.

DESIGN, PARTICIPANTS, MAIN MEASURES

Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend.

KEY RESULTS

The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications).

CONCLUSIONS

The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.

Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback

BACKGROUND

Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability.

OBJECTIVE

Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool.

DESIGN, PARTICIPANTS, AND MAIN MEASURES

The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting.

KEY RESULTS

The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality.

CONCLUSIONS

The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.

The use of podcasts as a tool to teach clinical reasoning: a pseudorandomized and controlled study

OBJECTIVES

Podcasts have emerged as an efficient method for widespread delivery of educational clinical reasoning (CR) content. However, the impact of such podcasts on CR skills has not been established. We set out to determine whether exposure to expert reasoning in a podcast format leads to enhanced CR skills.

METHODS

This is a pseudo-randomized study of third-year medical students (MS3) to either a control group (n=22) of pre-established online CR modules, or intervention group (n=26) with both the online modules and novel CR podcasts. The podcasts were developed from four "clinical unknown" cases presented to expert clinician educators. After completing these assignments in weeks 1-2, weekly history and physical (H&P) notes were collected and graded according to the validated IDEA rubric between weeks 3-7. A longitudinal regression model was used to compare the H&P IDEA scores over time. Usage and perception of the podcasts was also assessed via survey data.

RESULTS

Ninety control and 128 intervention H&Ps were scored. There was no statistical difference in the change of average IDEA scores between intervention (0.92, p=0.35) and control groups (-0.33, p=0.83). Intervention participants positively received the podcasts and noted increased discussion of CR principles from both their ward (3.1 vs. 2.4, p=0.08) and teaching (3.2 vs. 2.5, p=0.05) attendings.

CONCLUSIONS

This is the first objective, pseudo-randomized assessment of CR podcasts in undergraduate medical education. While we did not demonstrate significant improvement in IDEA scores, our data show that podcasts are a well-received tool that can prime learners to recognize CR principles.

Evaluating the Clinical Reasoning of Student Health Professionals in Placement and Simulation Settings: A Systematic Review

(1) Background: Clinical reasoning is essential to the effective practice of autonomous health professionals and is, therefore, an essential capability to develop as students. This review aimed to systematically identify the tools available to health professional educators to evaluate students' attainment of clinical reasoning capabilities in clinical placement and simulation settings. (2) Methods: A systemic review of seven databases was undertaken. Peer-reviewed, English-language publications reporting studies that developed or tested relevant tools were included. Searches included multiple terms related to clinical reasoning and health disciplines. Data regarding each tool's conceptual basis and evaluated constructs were systematically extracted and analysed. (3) Results: Most of the 61 included papers evaluated students in medical and nursing disciplines, and over half reported on the Script Concordance Test or Lasater Clinical Judgement Rubric. A number of conceptual frameworks were referenced, though many papers did not reference any framework. (4) Conclusions: Overall, key outcomes highlighted an emphasis on diagnostic reasoning, as opposed to management reasoning. Tools were predominantly aligned with individual health disciplines and with limited cross-referencing within the field. Future research into clinical reasoning evaluation tools should build on and refer to existing approaches and consider contributions across professional disciplinary divides.

Assessing the quality of electronic medical records as a platform for resident education

BACKGROUND

Previous studies have assessed note quality and the use of electronic medical record (EMR) as a part of medical training. However, a generalized and user-friendly note quality assessment tool is required for quick clinical assessment. We held a medical record writing competition and developed a checklist for assessing the note quality of participants' medical records. Using the checklist, this study aims to explore note quality between residents of different specialties and offer pedagogical implications.

METHODS

The authors created an inpatient checklist that examined fundamental EMR requirements through six note types and twenty items. A total of 149 records created by residents from 32 departments/stations were randomly selected. Seven senior physicians rated the EMRs using a checklist. Medical records were grouped as general medicine, surgery, paediatric, obstetrics and gynaecology, and other departments. The overall and group performances were analysed using analysis of variance (ANOVA).

RESULTS

Overall performance was rated as fair to good. Regarding the six note types, discharge notes (0.81) gained the highest scores, followed by admission notes (0.79), problem list (0.73), overall performance (0.73), progress notes (0.71), and weekly summaries (0.66). Among the five groups, other departments (80.20) had the highest total score, followed by obstetrics and gynaecology (78.02), paediatrics (77.47), general medicine (75.58), and surgery (73.92).

CONCLUSIONS

This study suggested that duplication in medical notes and the documentation abilities of residents affect the quality of medical records in different departments. Further research is required to apply the insights obtained in this study to improve the quality of notes and, thereby, the effectiveness of resident training.

Psychometric validation of the reconstructed version of the assessment of reasoning tool

BACKGROUND

Assessing learners' competence in diagnostic reasoning is challenging and unstandardized in medical education. We developed a theory-informed, behaviorally anchored rubric, the Assessment of Reasoning Tool (ART), with content and response process validity. This study gathered evidence to support the internal structure and the interpretation of measurements derived from this tool.

METHODS

We derived a reconstructed version of ART (ART-R) as a 15-item, 5-point Likert scale using the ART domains and descriptors. A psychometric evaluation was performed. We created 18 video variations of learner oral presentations, portraying different performance levels of the ART-R.

RESULTS

152 faculty viewed two videos and rated the learner globally and then using the ART-R. The confirmatory factor analysis showed a favorable comparative fit index = 0.99, root mean square error of approximation = 0.097, and standardized root mean square residual = 0.026. The five domains, hypothesis-directed information gathering, problem representation, prioritized differential diagnosis, diagnostic evaluation, and awareness of cognitive tendencies/emotional factors, had high internal consistency. The total score for each domain had a positive association with the global assessment of diagnostic reasoning.

CONCLUSIONS

Our findings provide validity evidence for the ART-R as an assessment tool with five theoretical domains, internal consistency, and association with global assessment.

Assessing Clinical Reasoning: Targeting the Higher Levels of the Pyramid

Clinical reasoning is a core component of clinical competency that is used in all patient encounters from simple to complex presentations. It involves synthesis of myriad clinical and investigative data, to generate and prioritize an appropriate differential diagnosis and inform safe and targeted management plans.The literature is rich with proposed methods to teach this critical skill to trainees of all levels. Yet, ensuring that reasoning ability is appropriately assessed across the spectrum of knowledge acquisition to workplace-based clinical performance can be challenging.In this perspective, we first introduce the concepts of illness scripts and dual-process theory that describe the roles of analytic system 1 and non-analytic system 2 reasoning in clinical decision making. Thereafter, we draw upon existing evidence and expert opinion to review a range of methods that allow for effective assessment of clinical reasoning, contextualized within Miller's pyramid of learner assessment. Key assessment strategies that allow teachers to evaluate their learners' clinical reasoning ability are described from the level of knowledge acquisition, through to real-world demonstration in the clinical workplace.

Implementation of a clinical reasoning curriculum for clerkship-level medical students: a pseudo-randomized and controlled study

Background The National Academies of Sciences report Improving Diagnosis in Healthcare highlighted the need for better training in medical decision-making, but most medical schools lack formal education in clinical reasoning. Methods We conducted a pseudo-randomized and controlled study to evaluate the impact of a clinical reasoning curriculum in an internal medicine clerkship. Students in the intervention group completed six interactive online modules focused on reasoning concepts and a skills-based workshop. We assessed the impact of the curriculum on clinical reasoning knowledge and skills and perception of education by evaluating: (1) performance on a clinical reasoning concept quiz, (2) demonstration of reasoning in hospital admission notes, and (3) awareness of attending physician utilization of clinical reasoning concepts. Results Students in the intervention group demonstrated superior performance on the clinical reasoning knowledge quiz (67% vs. 54%, p < 0.001). Students in the intervention group demonstrated superior written reasoning skills in the data synthesis (2.3 vs. 2.0, p = 0.02) and diagnostic reasoning (2.2 vs. 1.9, p = 0.02) portions of their admission notes, and reported more discussion of clinical reasoning by their attending physicians. Conclusions Exposure to a clinical reasoning curriculum was associated with superior reasoning knowledge and superior written demonstration of clinical reasoning skills by third-year medical students on an internal medicine clerkship.

Clinical Reasoning Assessment Methods: A Scoping Review and Practical Guidance

PURPOSE

An evidence-based approach to assessment is critical for ensuring the development of clinical reasoning (CR) competence. The wide array of CR assessment methods creates challenges for selecting assessments fit for the purpose; thus, a synthesis of the current evidence is needed to guide practice. A scoping review was performed to explore the existing menu of CR assessments.

METHOD

Multiple databases were searched from their inception to 2016 following PRISMA guidelines. Articles of all study design types were included if they studied a CR assessment method. The articles were sorted by assessment methods and reviewed by pairs of authors. Extracted data were used to construct descriptive appendixes, summarizing each method, including common stimuli, response formats, scoring, typical uses, validity considerations, feasibility issues, advantages, and disadvantages.

RESULTS

A total of 377 articles were included in the final synthesis. The articles broadly fell into three categories: non-workplace-based assessments (e.g., multiple-choice questions, extended matching questions, key feature examinations, script concordance tests); assessments in simulated clinical environments (objective structured clinical examinations and technology-enhanced simulation); and workplace-based assessments (e.g., direct observations, global assessments, oral case presentations, written notes). Validity considerations, feasibility issues, advantages, and disadvantages differed by method.

CONCLUSIONS

There are numerous assessment methods that align with different components of the complex construct of CR. Ensuring competency requires the development of programs of assessment that address all components of CR. Such programs are ideally constructed of complementary assessment methods to account for each method's validity and feasibility issues, advantages, and disadvantages.

The Assessment of Reasoning Tool (ART): structuring the conversation between teachers and learners

Background

Excellence in clinical reasoning is one of the most important outcomes of medical education programs, but assessing learners’ reasoning to inform corrective feedback is challenging and unstandardized.

Methods

The Society to Improve Diagnosis in Medicine formed a multi-specialty team of medical educators to develop the Assessment of Reasoning Tool (ART). This paper describes the tool development process. The tool was designed to facilitate clinical teachers’ assessment of learners’ oral presentation for competence in clinical reasoning and facilitate formative feedback. Reasoning frameworks (e.g. script theory), contemporary practice goals (e.g. high-value care [HVC]) and proposed error reduction strategies (e.g. metacognition) were used to guide the development of the tool.

Results

The ART is a behaviorally anchored, three-point scale assessing five domains of reasoning: (1) hypothesis-directed data gathering, (2) articulation of a problem representation, (3) formulation of a prioritized differential diagnosis, (4) diagnostic testing aligned with HVC principles and (5) metacognition. Instructional videos were created for faculty development for each domain, guided by principles of multimedia learning.

Conclusions

The ART is a theory-informed assessment tool that allows teachers to assess clinical reasoning and structure feedback conversations.

Validity Evidence for a Residency Admissions Standardized Assessment Letter for Pediatrics

Construct: This study aims to provide validity evidence for the standardized Assessment Letter for Pediatrics as a measure of competencies expected of a 1st-year pediatrics resident as part of a pediatric residency admissions process.

BACKGROUND

The Narrative Letter of Recommendation is a frequently used tool in the residency admissions process even though it has poor interrater reliability, lacks pertinent content, and does not correlate with residency performance. A newer tool, the Standardized Letter, has shown validity evidence for content and interrater reliability in other specialties. We sought to develop and provide validity evidence for the standardized Assessment Letter for Pediatrics.

APPROACH

All 2012 and 2013 applicants invited to interview at the University of Geneva Pediatrics Residency Program provided 2 standardized Assessment Letters. Content for the letter was based on CanMEDS roles and ratings of 6 desired competencies and an overall assessment. Validity evidence was gathered for internal structure (Cronbach's alpha and generalizability), response process (interrater reliability with intraclass correlation), relations to other variables (Pearson's correlation coefficient), and consequences (logistic regression to predict admission).

RESULTS

One hundred fourteen faculty completed 142 standardized Assessment Letters for 71 applicants. Average overall assessment was 3.0 of 4 (SD = 0.59). Cronbach's alpha was 0.93. The G-coefficient was 0.59. The decision study projected that four Assessment Letters are needed to attain a G-coefficient of 0.73. Applicant variance (28.5%) indicated high applicant differentiation. The Assessment Letter intraclass coefficient was 0.51, 95% confidence interval (CI) [0.43, 0.59]. Assessment Letter scores were correlated with the structured interview (r = .28), 95% CI [0.05, 0.51]; global rating (r = .36), 95% CI [0.13, 0.58]; and admissions decision (r = .25), 95% CI [0.02, 0.46]. Assessment Letter scores did not predict the admissions decision (odds ratio = 1.67, p = .37) after controlling for the unique contribution of the structured interview and global rating scores.

CONCLUSION

Validity evidence supports use of the Assessment Letter for Pediatrics; future studies should refine items to improve predictive validity and explore how to best integrate the Assessment Letter into the residency admissions process.

Analyzing the effectiveness of teaching and factors in clinical decision-making

Objective

The aim of this study is to prepare junior physicians, clinical education should focus on the teaching of clinical decision-making. This research is designed to explore teaching of clinical decision-making and to analyze the benefits of an "Analogy guide clinical decision-making" as a learning intervention for junior doctors.

Materials and Methods

This study had a "quasi-experimental design" and was conducted in a medical center in eastern Taiwan. Participants and Program Description: Thirty junior doctors and three clinical teachers were involved in the study. The experimental group (15) received 1 h of instruction from the "Analogy guide for teaching clinical decision-making" every day for 3 months. Program Evaluation: A "Clinical decision-making self-evaluation form" was used as the assessment tool to evaluate participant learning efficiency before and after the teaching program. Semi-structured qualitative research interviews were also conducted.

Results

We found using the analogy guide for teaching clinical decision-making could help enhance junior doctors' self-confidence. Important factors influencing clinical decision-making included workload, decision-making, and past experience.

Conclusion

Clinical teaching using the analogy guide for clinical decision-making may be a helpful tool for training and can contribute to a more comprehensive understanding of decision-making.

Situational awareness within objective structured clinical examination stations in undergraduate medical training - a literature search

BACKGROUND

Medical students may not be able to identify the essential elements of situational awareness (SA) necessary for clinical reasoning. Recent studies suggest that students have little insight into cognitive processing and SA in clinical scenarios. Objective Structured Clinical Examinations (OSCEs) could be used to assess certain elements of situational awareness. The purpose of this paper is to review the literature with a view to identifying whether levels of SA based on Endsley's model can be assessed utilising OSCEs during undergraduate medical training.

METHODS

A systematic search was performed pertaining to SA and OSCEs, to identify studies published between January 1975 (first paper describing an OSCE) and February 2017, in peer reviewed international journals published in English. PUBMED, EMBASE, PsycINFO Ovid and SCOPUS were searched for papers that described the assessment of SA using OSCEs among undergraduate medical students. Key search terms included "objective structured clinical examination", "objective structured clinical assessment" or "OSCE" and "non-technical skills", "sense-making", "clinical reasoning", "perception", "comprehension", "projection", "situation awareness", "situational awareness" and "situation assessment". Boolean operators (AND, OR) were used as conjunctions to narrow the search strategy, resulting in the limitation of papers relevant to the research interest. Areas of interest were elements of SA that can be assessed by these examinations.

RESULTS

The initial search of the literature retrieved 1127 publications. Upon removal of duplicates and papers relating to nursing, paramedical disciplines, pharmacy and veterinary education by title, abstract or full text, 11 articles were eligible for inclusion as related to the assessment of elements of SA in undergraduate medical students.

DISCUSSION

Review of the literature suggests that whole-task OSCEs enable the evaluation of SA associated with clinical reasoning skills. If they address the levels of SA, these OSCEs can provide supportive feedback and strengthen educational measures associated with higher diagnostic accuracy and reasoning abilities.

CONCLUSION

Based on the findings, the early exposure of medical students to SA is recommended, utilising OSCEs to evaluate and facilitate SA in dynamic environments.

Self-Directed Rater Training for Pediatric History and Physical Exam Evaluation (P-HAPEE) Rubric, a Validated Written H&P Assessment Tool

INTRODUCTION

We developed, revised, and implemented self-directed rater training materials in the course of a validity study for a written Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric.

METHODS

Core training materials consist of a single-page instruction sheet, sample written history and physical (H&P), and detailed answer key. We iteratively revised the materials based on reviewer comments and pilot testing. Eighteen attending physicians and five senior residents underwent self-directed training, scored 10 H&Ps, and completed a rubric utility survey in the course of the validity study. We have since implemented the P-HAPEE rubric and self-directed rater training in a pediatric clerkship. Based on input from reviewers, study raters, faculty members, and medical student users, we have also developed and implemented additional optional supplemental training materials.

RESULTS

Pilot testing indicated that training takes approximately 1 hour. While reviewers endorsed the training format, several suggested having optional supplemental materials available. Nineteen out of 23 volunteer study raters completed the rubric utility survey. All described the rubric as good or very good and indicated strong to very strong interest in continued use.

DISCUSSION

The P-HAPEE rubric offers a novel, practical, reliable, and valid method for supervising physicians to assess pediatric written H&Ps and can be implemented using brief, self-directed rater training.

Assessment of SOAP note evaluation tools in colleges and schools of pharmacy

INTRODUCTION

To describe current methods used to assess SOAP notes in colleges and schools of pharmacy.

METHODS

Members of the American Association of Colleges of Pharmacy Laboratory Instructors Special Interest Group were invited to share assessment tools for SOAP notes. Content of submissions was evaluated to characterize overall qualities and how the tools assessed subjective, objective, assessment, and plan information.

RESULTS

Thirty-nine assessment tools from 25 schools were evaluated. Twenty-nine (74%) of the tools were rubrics and ten (26%) were checklists. All rubrics included analytic scoring elements, while two (7%) were mixed with holistic and analytic scoring elements. A majority of the rubrics (35%) used a four-item rating scale. Substantial variability existed in how tools evaluated subjective and objective sections. All tools included problem identification in the assessment section. Other assessment items included goals (82%) and rationale (69%). Seventy-seven percent assessed drug therapy; however, only 33% assessed non-drug therapy. Other plan items included education (59%) and follow-up (90%).

DISCUSSION AND CONCLUSIONS

There is a great deal of variation in the specific elements used to evaluate SOAP notes in colleges and schools of pharmacy. Improved consistency in assessment methods to evaluate SOAP notes may better prepare students to produce standardized documentation when entering practice.

Developing Validity Evidence for the Written Pediatric History and Physical Exam Evaluation Rubric

OBJECTIVE

The written history and physical examination (H&P) is an underutilized source of medical trainee assessment. The authors describe development and validity evidence for the Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric: a novel tool for evaluating written H&Ps.

METHODS

Using an iterative process, the authors drafted, revised, and implemented the 10-item rubric at 3 academic institutions in 2014. Eighteen attending physicians and 5 senior residents each scored 10 third-year medical student H&Ps. Inter-rater reliability (IRR) was determined using intraclass correlation coefficients. Cronbach α was used to report consistency and Spearman rank-order correlations to determine relationships between rubric items. Raters provided a global assessment, recorded time to review and score each H&P, and completed a rubric utility survey.

RESULTS

Overall intraclass correlation was 0.85, indicating adequate IRR. Global assessment IRR was 0.89. IRR for low- and high-quality H&Ps was significantly greater than for medium-quality ones but did not differ on the basis of rater category (attending physician vs. senior resident), note format (electronic health record vs nonelectronic), or student diagnostic accuracy. Cronbach α was 0.93. The highest correlation between an individual item and total score was for assessments was 0.84; the highest interitem correlation was between assessment and differential diagnosis (0.78). Mean time to review and score an H&P was 16.3 minutes; residents took significantly longer than attending physicians. All raters described rubric utility as "good" or "very good" and endorsed continued use.

CONCLUSIONS

The P-HAPEE rubric offers a novel, practical, reliable, and valid method for supervising physicians to assess pediatric written H&Ps.