1
|
Lees AF, Beni C, Lee A, Wedgeworth P, Dzara K, Joyner B, Tarczy-Hornoch P, Leu M. Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:1326-1336. [PMID: 37267042 PMCID: PMC10615720 DOI: 10.1097/acm.0000000000005288] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
PURPOSE This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies. METHOD In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency. RESULTS The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment. CONCLUSIONS This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.
Collapse
Affiliation(s)
- A Fischer Lees
- A. Fischer Lees is a clinical informatics fellow, Department of Biomedical Informatics and Medical Education, University of Washington School of Medicine, Seattle, Washington
| | - Catherine Beni
- C. Beni is a general surgery resident, Department of Surgery, University of Washington School of Medicine, Seattle, Washington
| | - Albert Lee
- A. Lee is a clinical informatics fellow, Department of Biomedical Informatics and Medical Education, University of Washington School of Medicine, Seattle, Washington
| | - Patrick Wedgeworth
- P. Wedgeworth is a clinical informatics fellow, Department of Biomedical Informatics and Medical Education, University of Washington School of Medicine, Seattle, Washington
| | - Kristina Dzara
- K. Dzara is assistant dean for educator development, director, Center for Learning and Innovation in Medical Education, and associate professor of medical education, Department of Biomedical Informatics and Medical Education, University of Washington School of Medicine, Seattle, Washington
| | - Byron Joyner
- B. Joyner is vice dean for graduate medical education and a designated institutional official, Graduate Medical Education, University of Washington School of Medicine, Seattle, Washington
| | - Peter Tarczy-Hornoch
- P. Tarczy-Hornoch is professor and chair, Department of Biomedical Informatics and Medical Education, and professor, Department of Pediatrics (Neonatology), University of Washington School of Medicine, and adjunct professor, Allen School of Computer Science and Engineering, University of Washington, Seattle, Washington
| | - Michael Leu
- M. Leu is professor and director, Clinical Informatics Fellowship, Department of Biomedical Informatics and Medical Education, and professor, Department of Pediatrics, University of Washington School of Medicine, Seattle, Washington
| |
Collapse
|
2
|
Mai MV, Orenstein EW, Manning JD, Luberti AA, Dziorny AC. Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs. Appl Clin Inform 2020; 11:442-451. [PMID: 32583389 DOI: 10.1055/s-0040-1713133] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023] Open
Abstract
OBJECTIVE Patient attribution, or the process of attributing patient-level metrics to specific providers, attempts to capture real-life provider-patient interactions (PPI). Attribution holds wide-ranging importance, particularly for outcomes in graduate medical education, but remains a challenge. We developed and validated an algorithm using EHR data to identify pediatric resident PPIs (rPPIs). METHODS We prospectively surveyed residents in three care settings to collect self-reported rPPIs. Participants were surveyed at the end of primary care clinic, emergency department (ED), and inpatient shifts, shown a patient census list, asked to mark the patients with whom they interacted, and encouraged to provide a short rationale behind the marked interaction. We extracted routine EHR data elements, including audit logs, note contribution, order placement, care team assignment, and chart closure, and applied a logistic regression classifier to the data to predict rPPIs in each care setting. We also performed a comment analysis of the resident-reported rationales in the inpatient care setting to explore perceived patient interactions in a complicated workflow. RESULTS We surveyed 81 residents over 111 shifts and identified 579 patient interactions. Among EHR extracted data, time-in-chart was the best predictor in all three care settings (primary care clinic: odds ratio [OR] = 19.36, 95% confidence interval [CI]: 4.19-278.56; ED: OR = 19.06, 95% CI: 9.53-41.65' inpatient: OR = 2.95, 95% CI: 2.23-3.97). Primary care clinic and ED specific models had c-statistic values > 0.98, while the inpatient-specific model had greater variability (c-statistic = 0.89). Of 366 inpatient rPPIs, residents provided rationales for 90.1%, which were focused on direct involvement in a patient's admission or transfer, or care as the front-line ordering clinician (55.6%). CONCLUSION Classification models based on routinely collected EHR data predict resident-defined rPPIs across care settings. While specific to pediatric residents in this study, the approach may be generalizable to other provider populations and scenarios in which accurate patient attribution is desirable.
Collapse
Affiliation(s)
- Mark V Mai
- Department of Anesthesia and Critical Care Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States.,Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Evan W Orenstein
- Department of Pediatrics, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
| | - John D Manning
- Department of Emergency Medicine, Atrium Health's Carolinas Medical Center, Charlotte, North Carolina, United States
| | - Anthony A Luberti
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States.,Department of Pediatrics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Adam C Dziorny
- Department of Anesthesia and Critical Care Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States.,Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| |
Collapse
|
3
|
Li J, Roosevelt G, McCabe K, Preotle J, Pereira F, Takayesu JK, Monuteaux M, Bachur RG. Pediatric Case Exposure During Emergency Medicine Residency. AEM EDUCATION AND TRAINING 2018; 2:317-327. [PMID: 30386842 PMCID: PMC6194046 DOI: 10.1002/aet2.10130] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/17/2018] [Revised: 08/28/2018] [Accepted: 09/04/2018] [Indexed: 05/12/2023]
Abstract
OBJECTIVE While emergency medicine (EM) physicians treat the majority of pediatric EM (PEM) patients in the United States, little is known about their PEM experience during training. The primary objective was to characterize the pediatric case exposure and compare to established EM residency training curricula among EM residents across five U.S. residency programs. METHODS We performed a multicenter medical record review of all pediatric patients (aged < 18 years) seen by the 2015 graduating resident physicians at five U.S. EM training programs. Resident-level counts of pediatric patients were measured and specific counts were classified by the 2016 Model of Clinical Practice of Emergency Medicine (MCP) and Pediatric Emergency Care Applied Research Network (PECARN) diagnostic categories. We assessed variability between residents and between programs. RESULTS A total of 36,845 children were managed by 68 residents across all programs. The median age was 6 years. The median number of patients per resident was 660 with an interquartile range of 336. The most common PECARN diagnostic categories were trauma, gastrointestinal, and respiratory disease. Thirty-two core MCP diagnoses (43% of MCP list) were not seen by at least 50% of the residents. We found statistically significant variability between programs in both PECARN diagnostic categories (p < 0.01) and MCP diagnoses (p < 0.01). CONCLUSION There is considerable variation in the number of pediatric patients and the diagnostic case volume seen by EM residents. The relationship between this case variability and competence upon graduation is unknown; further investigation is warranted to better inform program-specific curricula and to guide training requirements in EM.
Collapse
Affiliation(s)
- Joyce Li
- Division of Emergency MedicineBoston Children's HospitalBostonMA
- Department of Emergency Medicine and PediatricsHarvard Medical SchoolBostonMA
| | - Genie Roosevelt
- Denver Health Medical CenterDepartment of Emergency MedicineUniversity of Colorado School of MedicineDenverCO
| | - Kerry McCabe
- Boston Medical CenterDepartment of Emergency MedicineBoston University School of MedicineBostonMA
| | - Jane Preotle
- Hasbro Children's HospitalDepartment of Emergency MedicineAlpert Medical School of Brown UniversityProvidenceRI
| | - Faria Pereira
- Texas Children's HospitalDivision of Emergency MedicineBaylor College of MedicineHoustonTX
| | - James K. Takayesu
- Massachusetts General HospitalDivision of Emergency MedicineHarvard Medical SchoolBostonMA
| | - Michael Monuteaux
- Division of Emergency MedicineBoston Children's HospitalBostonMA
- Department of Emergency Medicine and PediatricsHarvard Medical SchoolBostonMA
| | - Richard G. Bachur
- Division of Emergency MedicineBoston Children's HospitalBostonMA
- Department of Emergency Medicine and PediatricsHarvard Medical SchoolBostonMA
| |
Collapse
|
4
|
Closed reduction of shoulder dislocations in the emergency department: does an emergency medicine residency affect the orthopaedic resident experience? CURRENT ORTHOPAEDIC PRACTICE 2017. [DOI: 10.1097/bco.0000000000000564] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
5
|
Abstract
BACKGROUND The Accreditation Council for Graduate Medical Education has called for increased emphasis on reporting objective performance measures to trainees and programs. However, reporting of objective measures, including clinical volume, is largely omitted from training programs. OBJECTIVE To use automated electronic medical systems at a tertiary pediatric care hospital to create a dashboard that reports objective trainee and program metrics, including clinical volume and diagnoses in a pediatrics residency. METHODS We queried an enterprise data warehouse that aggregates data daily from multiple hospital systems to identify patient encounters during which senior pediatrics residents at Boston Children's Hospital had entered documentation over a 9-month period. From this query, we created a filterable dashboard to display clinical volume and diagnosis data by individual resident and in aggregate. RESULTS A total of 44 of 45 senior residents (98%) in the program were included in analysis. We identified 12 198 patient encounters during which a senior pediatrics resident had entered documentation; these included a median of 332 inpatient encounters per resident, 122 emergency department encounters, and 84 outpatient encounters. The most common diagnoses stratified by clinical site were: inpatient - dehydration (median = 61); emergency department - long-term/current drug therapy (median = 16); and outpatient - encounter for immunization (median = 48). CONCLUSIONS We used electronic health record systems to generate performance dashboards for trainees in a pediatrics residency across different sites of care with reported volume by diagnosis. Our dashboards provide feedback to program leadership regarding individual and aggregate trainee experience and allow individual trainees to compare their clinical exposure to peers.
Collapse
|
6
|
Rajkomar A, Ranji SR, Sharpe B. Using the Electronic Health Record to Identify Educational Gaps for Internal Medicine Interns. J Grad Med Educ 2017; 9:109-112. [PMID: 28261404 PMCID: PMC5319608 DOI: 10.4300/jgme-d-16-00272.1] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/26/2023] Open
Abstract
BACKGROUND An important component of internal medicine residency is clinical immersion in core rotations to expose first-year residents to common diagnoses. OBJECTIVE Quantify intern experience with common diagnoses through clinical documentation in an electronic health record. METHODS We analyzed all clinical notes written by postgraduate year (PGY) 1, PGY-2, and PGY-3 residents on medicine service at an academic medical center July 1, 2012, through June 30, 2014. We quantified the number of notes written by PGY-1s at 1 of 3 hospitals where they rotate, by the number of notes written about patients with a specific principal billing diagnosis, which we defined as diagnosis-days. We used the International Classification of Diseases 9 (ICD-9) and the Clinical Classification Software (CCS) to group the diagnoses. RESULTS We analyzed 53 066 clinical notes covering 10 022 hospitalizations with 1436 different ICD-9 diagnoses spanning 217 CCS diagnostic categories. The 10 most common ICD-9 diagnoses accounted for 23% of diagnosis-days, while the 10 most common CCS groupings accounted for more than 40% of the diagnosis-days. Of 122 PGY-1s, 107 (88%) spent at least 2 months on the service, and 3% were exposed to all of the top 10 ICD-9 diagnoses, while 31% had experience with fewer than 5 of the top 10 diagnoses. In addition, 17% of PGY-1s saw all top 10 CCS diagnoses, and 5% had exposure to fewer than 5 CCS diagnoses. CONCLUSIONS Automated detection of clinical experience may help programs review inpatient clinical experiences of PGY-1s.
Collapse
Affiliation(s)
- Alvin Rajkomar
- Corresponding author: Alvin Rajkomar, MD, University of California, San Francisco, 533 Parnassus Avenue, Suite 127A, San Francisco, CA 94143-0131, 415.476.1528, fax 415.502.1976,
| | | | | |
Collapse
|
7
|
Mattana J, Kerpen H, Lee C, Multz A, Pekmezaris R, Napolitano B, Walia R, Steinberg H. Quantifying internal medicine resident clinical experience using resident-selected primary diagnosis codes. J Hosp Med 2011; 6:395-400. [PMID: 21916001 DOI: 10.1002/jhm.892] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
BACKGROUND While experiential learning is a desirable goal of residency education, little is known regarding the actual clinical experience of internal medicine residents during their training. METHODS We modified an electronic patient handoff tool to include a system for resident entry of a primary diagnosis for each of their patients. Using the International Classification of Diseases, Ninth Revision (ICD-9) system, we created two methods to select the code: 1) an organ system-based dropdown list containing frequently used codes; and 2) a search option for the complete ICD-9 database. The codes were then grouped using ICD-9 categorization. RESULTS A total of 7562 resident-patient diagnostic encounters were studied. A wide spectrum of clinical conditions was observed, with symptoms and ill-defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/immune disorders, and disorders of the skin and subcutaneous tissue accounting for about 86% of resident clinical experience. Symptoms and ill-defined conditions were noted to represent a sizable portion of resident clinical experience. Within this category, the most common conditions were fever; abdominal pain; and chest pain, unspecified. CONCLUSIONS Analysis of resident-selected ICD-9 codes might serve as a method to attempt to define resident clinical experience, and may be useful in the development of innovative experiential learning-based residency curricula. This might also be used to assess gaps in experiential learning at the program or resident level, and may serve to identify topics that require additional teaching supplementation.
Collapse
Affiliation(s)
- Joseph Mattana
- Department of Medicine, Long Island Jewish Medical Center, North Shore-Long Island Jewish Health System, Great Neck, New York, USA.
| | | | | | | | | | | | | | | |
Collapse
|
8
|
Simpao A, Heitz JW, McNulty SE, Chekemian B, Brenn BR, Epstein RH. The Design and Implementation of an Automated System for Logging Clinical Experiences Using an Anesthesia Information Management System. Anesth Analg 2011; 112:422-9. [DOI: 10.1213/ane.0b013e3182042e56] [Citation(s) in RCA: 23] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
9
|
Use of an automated electronic case log to assess fellowship training: tracking the pediatric emergency medicine experience. Pediatr Emerg Care 2008; 24:75-82. [PMID: 18277842 DOI: 10.1097/pec.0b013e318163db3c] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
Abstract
The Accreditation Council for Graduate Medical Education has mandated the assessment of medical training across 6 core competencies. The patient care competency is at the core of medical training. With the introduction of patient tracking systems used in emergency departments, patient-physician encounters can be systematically studied. The combination of tracking data with other clinical information systems can be used to create an electronic case log to quantify the experience of fellows, thereby offering a summative measure of the patient care competency. We used an automated case log to assess clinical exposure in our pediatric emergency medicine fellowship.
Collapse
|