The Design and Implementation of an Automated System for Logging Clinical Experiences Using an Anesthesia Information Management System

Current Procedural Terminology Code Selection, Attitudes, and Practices of the Orthopaedic Surgery Resident Case Log: A Survey of Residents and Program Directors

Introduction

The Accreditation Council for Graduate Medical Education Resident Case Log is one of the primary tools used to track surgical experience. Owing to the self-reported nature of case logging, there is uncertainty in the consistency and accuracy of case logging. The aims of this study are two-fold: to assess current resident case log Current Procedural Terminology (CPT) code selection and practices across orthopaedic surgery residencies and to understand current attitudes of both program directors (PD) and residents surrounding case logging.

Methods

Residents and PDs from 18 residency programs received standardized, consensus-built surveys distributed through the Collaborative Orthopaedic Educational Research Group. Resident surveys additionally contained clinical orthopaedic subspecialties vignettes on sports, trauma, and spine. Each subspecialty section contained 4 clinical vignettes with stepwise increases in complexity/CPT coding procedures.

Results

One hundred sixteen residents (response rate: 28.4%) and 16 PDs (response rate: 88.9%) participated. Formal case log training was reported by 53.0% of residents and 56.3% of PDs. A total of 7.8% of residents rated themselves "excellent" at applying CPT codes for the case log, while 0.0% PDs rated their residents' ability as "excellent." In total, 40.9% of residents and 81.3% of PDs responded that it was "extremely important" or "very important" to code accurately (p = 0.006). Agreement between resident CPT code selection and number of cases and procedures logged for each clinical vignette was conducted using Fleiss' kappa. As the clinical vignettes increased in complexity, there was a decreasing trend in kappa values from the first (least complex) to the last (most complex) clinical vignette.

Conclusions

The inconsistent case logging practices, dubious outlook on case log accuracy and resident case logging ability and attitude, and lack of formal training signals a need for formal, standardized case log training. Enhanced case logging instruction and formalized educational training for PDs and residents would be a meaningful step toward capturing true operative experience, which would have a substantial impact on orthopaedic surgery resident education and assessment.

Uses of Electronic Health Record Data to Measure the Clinical Learning Environment of Graduate Medical Education Trainees: A Systematic Review

PURPOSE

This study systematically reviews the uses of electronic health record (EHR) data to measure graduate medical education (GME) trainee competencies.

METHOD

In January 2022, the authors conducted a systematic review of original research in MEDLINE from database start to December 31, 2021. The authors searched for articles that used the EHR as their data source and in which the individual GME trainee was the unit of observation and/or unit of analysis. The database query was intentionally broad because an initial survey of pertinent articles identified no unifying Medical Subject Heading terms. Articles were coded and clustered by theme and Accreditation Council for Graduate Medical Education (ACGME) core competency.

RESULTS

The database search yielded 3,540 articles, of which 86 met the study inclusion criteria. Articles clustered into 16 themes, the largest of which were trainee condition experience (17 articles), work patterns (16 articles), and continuity of care (12 articles). Five of the ACGME core competencies were represented (patient care and procedural skills, practice-based learning and improvement, systems-based practice, medical knowledge, and professionalism). In addition, 25 articles assessed the clinical learning environment.

CONCLUSIONS

This review identified 86 articles that used EHR data to measure individual GME trainee competencies, spanning 16 themes and 6 competencies and revealing marked between-trainee variation. The authors propose a digital learning cycle framework that arranges sequentially the uses of EHR data within the cycle of clinical experiential learning central to GME. Three technical components necessary to unlock the potential of EHR data to improve GME are described: measures, attribution, and visualization. Partnerships between GME programs and informatics departments will be pivotal in realizing this opportunity.

Implementation and Evaluation of Integrating an Electronic Health Record With the ACGME Case Log System

BACKGROUND

It is essential to log resident-performed procedures to assess training programs and fulfill specialty requirements, but resident case numbers are often underreported. Current systems require inefficient data entry steps, and residents and fellows report that user interfaces and administrative burden contribute to logging inaccuracy.

OBJECTIVE

To determine the accuracy, feasibility, and acceptability of a single logging approach for resident case logging.

METHODS

In 2018, we implemented a case logging system integrated with the institutional electronic health record (EHR) and the Accreditation Council for Graduate Medical Education (ACGME) case log system to record procedures performed by ophthalmology residents. We compared the proportion of resident-performed cataract extractions in the EHR that were reported to ACGME for 3 periods: before the deployment of the new system (6 months), during the transition (6 months), and after the change (2 years). Resident satisfaction with the new system was evaluated using surveys.

RESULTS

An analysis of resident cataract surgeries showed that the percentage of resident cases logged increased from 85% prior to implementation to 91% after implementation. The integrated system became the preferred case logging method, with 100% of all logged cases being entered using the new platform. Surveys showed that the percentage of trainees who were moderately or very satisfied with the case log process increased from 55% before implementation to 100% after implementation.

CONCLUSIONS

A resident case log system integrated with an EHR more accurately reflects resident operative volume and increases trainee satisfaction with the logging process.

Automated Procedure Logs for Cardiology Fellows: A New Training Paradigm in the Era of Electronic Health Records

BACKGROUND

Procedural experience for residents and fellows is critical for achieving competence, and documentation of procedures performed is required. Procedure logs serve as the record of this experience, but are commonly generated manually, require substantial administrative effort, and cannot be corroborated for accuracy.

OBJECTIVE

We developed and implemented a structured clinical-educational report template (CERT), which automatically generates procedure logs directly from the clinical record.

METHODS

Our CERT aimed to replace the post-procedure note template for our cardiac catheterization laboratory and was incorporated into the electronic health record system. Numbers of documented procedures in automated CERT-derived procedure logs over a 1-year post-intervention period (2018-2019) were compared to manual logs and corrected for clinical volume changes. The CERT's impact on fellowship experience was also assessed.

RESULTS

Automated CERT procedure logs increased weekly procedural documentation over manual procedure logs for total procedures (24.2 ± 6.1 vs 17.1 ± 6.8, P = .007), left heart catheterizations (14.5 ± 3.6 vs 10.8 ± 4.2, P = .039), total procedural elements (40.2 ± 11.4 versus 20.9 ± 12.5, P < .001), and captured procedural details not previously documented. The CERT also reduced self-reported administrative time and improved fellowship experience.

CONCLUSIONS

A novel CERT allows procedure logs to be automatically derived from the clinical record and increased the number of documented procedures, compared to manual logging. This innovation ensures an accurate record of procedural experience and reduces self-reported non-educational administrative time for cardiology fellows.

Attributing Patients to Pediatric Residents Using Electronic Health Record Features Augmented with Audit Logs

OBJECTIVE

Patient attribution, or the process of attributing patient-level metrics to specific providers, attempts to capture real-life provider-patient interactions (PPI). Attribution holds wide-ranging importance, particularly for outcomes in graduate medical education, but remains a challenge. We developed and validated an algorithm using EHR data to identify pediatric resident PPIs (rPPIs).

METHODS

We prospectively surveyed residents in three care settings to collect self-reported rPPIs. Participants were surveyed at the end of primary care clinic, emergency department (ED), and inpatient shifts, shown a patient census list, asked to mark the patients with whom they interacted, and encouraged to provide a short rationale behind the marked interaction. We extracted routine EHR data elements, including audit logs, note contribution, order placement, care team assignment, and chart closure, and applied a logistic regression classifier to the data to predict rPPIs in each care setting. We also performed a comment analysis of the resident-reported rationales in the inpatient care setting to explore perceived patient interactions in a complicated workflow.

RESULTS

We surveyed 81 residents over 111 shifts and identified 579 patient interactions. Among EHR extracted data, time-in-chart was the best predictor in all three care settings (primary care clinic: odds ratio [OR] = 19.36, 95% confidence interval [CI]: 4.19-278.56; ED: OR = 19.06, 95% CI: 9.53-41.65' inpatient: OR = 2.95, 95% CI: 2.23-3.97). Primary care clinic and ED specific models had c-statistic values > 0.98, while the inpatient-specific model had greater variability (c-statistic = 0.89). Of 366 inpatient rPPIs, residents provided rationales for 90.1%, which were focused on direct involvement in a patient's admission or transfer, or care as the front-line ordering clinician (55.6%).

CONCLUSION

Classification models based on routinely collected EHR data predict resident-defined rPPIs across care settings. While specific to pediatric residents in this study, the approach may be generalizable to other provider populations and scenarios in which accurate patient attribution is desirable.

Case-Logging Practices in Otolaryngology Residency Training: National Survey of Residents and Program Directors

Objective (1) Evaluate the consistency and manner in which otolaryngology residents log surgical cases. (2) Assess the extent of instruction and guidance provided by program directors on case-logging practices. Study Design Cross-sectional national survey. Setting Accreditation Council for Graduate Medical Education otolaryngology residency programs in the United States. Subjects and Methods US otolaryngology residents, postgraduate year 2 through graduating chiefs as of July 2016, were recruited to respond to an anonymous questionnaire designed to characterize surgical case-logging practices. Program directors of US otolaryngology residency programs were recruited to respond to an anonymous questionnaire to elucidate how residents are instructed to log cases. Results A total of 272 residents and 53 program directors completed the survey, yielding response rates of 40.6% and 49.5%, respectively. Perceived accuracy of case logs is low among residents and program directors. Nearly 40% of residents purposely choose not to log certain cases, and 65.1% of residents underreport cases performed. More than 80% of program directors advise residents to log procedures performed outside the operating room, yet only 16% of residents consistently log such cases. Conclusion Variability in surgical case-logging behaviors and differences in provided instruction highlight the need for methods to improve consistency of logging practices. It is imperative to standardize practices across otolaryngology residency programs for case logs to serve as an accurate measure of surgical competency. This study provides a foundation for reform efforts within residency programs and for the Resident Case Log System.

Surgical Case Logging Habits and Attitudes: A Multispecialty Survey of Residents

OBJECTIVE

The Accreditation Council for Graduate Medical Education measures surgical residents' experience in the United States by mandating that residents log each procedure in which they have participated. This system is the primary mechanism by which breadth and depth of surgical training are documented, and data are used for program accreditation and by individual program directors to assess resident preparedness. The study objective was to learn from residents across surgical specialties how this system is being used, and whether they believe these data are reliable.

DESIGN

Investigators developed and administered a voluntary, 45-item survey. Resident demographic data, program details, logging behaviors, and attitudes were examined using descriptive statistics. Authors used multivariate logistic regression to assess respondent and program characteristics associated with logging habits.

SETTING

The survey was administered at a large academic medical center.

PARTICIPANTS

All general surgery, obstetrics and gynecology, orthopedics, urology, neurosurgery, otolaryngology, and plastic surgery residents were eligible. Of 126 surgical residents, 82 participated, yielding a response rate of 65%.

RESULTS

Overall, 7.5% considered the case log system highly inaccurate, 28.8% somewhat inaccurate, 52.5% somewhat accurate, and 11.3% highly accurate. Nearly half (48.1%) use an incorrect metric to log their role as surgeon or assistant. Half logged monthly or less frequently. The longest time residents reported falling behind ranged from less than a week to more than a year, with about half (51.4%) reporting backlogs of 3 months or longer. Approximately two-thirds considered the system difficult to navigate (64.2%) and burdensome (68.8%). Departmental training and reminders to log were associated with high fidelity logging habits.

CONCLUSIONS

Inconsistency of logging habits and perceived lack of accuracy raise concerns about use of the system for assessing surgical preparedness or accrediting training programs. Academic departments playing an active role may benefit from more reliable data to guide improvements in surgical training.

Paperless anesthesia: uses and abuses of these data

Demonstrably accurate records facilitate clinical decision making, improve patient safety, provide better defense against frivolous lawsuits, and enable better medical policy decisions. Anesthesia Information Management Systems (AIMS) have the potential to improve on the accuracy and reliability of handwritten records. Interfaces with electronic recording systems within the hospital or wider community allow correlation of anesthesia relevant data with biochemistry laboratory results, billing sections, radiological units, pharmacy, earlier patient records, and other systems. Electronic storage of large and accurate datasets has lent itself to quality assurance, enhancement of patient safety, research, cost containment, scheduling, anesthesia training initiatives, and has even stimulated organizational change. The time for record making may be increased by AIMS, but in some cases has been reduced. The question of impact on vigilance is not entirely settled, but substantial negative effects seem to be unlikely. The usefulness of these large databases depends on the accuracy of data and they may be incorrect or incomplete. Consequent biases are threats to the validity of research results. Data mining of biomedical databases makes it easier for individuals with political, social, or economic agendas to generate misleading research findings for the purpose of manipulating public opinion and swaying policymakers. There remains a fear that accessibility of data may have undesirable regulatory or legal consequences. Increasing regulation of treatment options during the perioperative period through regulated policies could reduce autonomy for clinicians. These fears are as yet unsubstantiated.

A Narrative Review of Meaningful Use and Anesthesia Information Management Systems

The US federal government has enacted legislation for a federal incentive program for health care providers and hospitals to implement electronic health records. The primary goal of the Meaningful Use (MU) program is to drive adoption of electronic health records nationwide and set the stage to monitor and guide efforts to improve population health and outcomes. The MU program provides incentives for the adoption and use of electronic health record technology and, in some cases, penalties for hospitals or providers not using the technology. The MU program is administrated by the Department of Health and Human Services and is divided into 3 stages that include specific reporting and compliance metrics. The rationale is that increased use of electronic health records will improve the process of delivering care at the individual level by improving the communication and allow for tracking population health and quality improvement metrics at a national level in the long run. The goal of this narrative review is to describe the MU program as it applies to anesthesiologists in the United States. This narrative review will discuss how anesthesiologists can meet the eligible provider reporting criteria of MU by applying anesthesia information management systems (AIMS) in various contexts in the United States. Subsequently, AIMS will be described in the context of MU criteria. This narrative literature review also will evaluate the evidence supporting the electronic health record technology in the operating room, including AIMS, independent of certification requirements for the electronic health record technology under MU in the United States.

General Surgery Resident Case Logs: Do They Accurately Reflect Resident Experience?

INTRODUCTION

Operative experience during residency lays the foundation for independent practice and additional specialty training following general surgery residency. The aim of this study was to examine operative experience of general surgery residents and detail the results of an intervention aimed at improving resident record keeping in the Accreditation Council for Graduate Medical Education (ACGME) case log system to better reflect their experience.

METHODS

Residents were asked to characterize variances in recorded operative experience identified through an audit of operative logs. Based on the results of the audit, an intervention was designed to prompt timely record keeping by residents. The intervention included education and discussion of survey audit results, weekly presentation of graphs detailing operative experience, and possible missed cases in the ACGME logs and addition of a first assistant column in morbidity and mortality (M&M) logs.

RESULTS

The audit of case logs identified discrepancies in 24.2% of the 636 cases examined. Chief residents were significantly more accurate (95.9%) in recording operative experience in ACGME case logs, whereas 50.3% of junior resident case logs contained variances. Residents characterized discrepancies as "forgot to log" (9.6%), "staff did the case" (5.2%), "another resident did more of the case" (3.6%), "other" (3.6%), a "more advanced resident was present for the case" (1.6%), "not present for case" (0.6%), and "left for consult" (0.3%). Over the 4-week intervention period, residents logged between 72.7% and 94.0% of cases. A month following the intervention period, we observed a 13.3% increase in recorded cases compared with the intervention period. Review of first assistant case logging following inclusion of a "first assistant" column in M&M logs demonstrated a 70.5% increase in first assistant cases logged into the ACGME system compared with the same time period a year ago.

CONCLUSIONS

Based on our results, we found that weekly displays of cases improved resident record keeping in the ACGME case log system, especially by junior residents. We believe that the addition of first assistant column on M&M lists, periodic audits reviewed at conferences, and semiannual evaluations will help junior residents more accurately report their experience during training.

Automated Case Cancellation Review System Improves Systems-Based Practice

BACKGROUND

Accreditation Council for Graduate Medical Education (ACGME) core competencies of systems-based practice and practice-based learning and improvement are difficult to assess, as they are often not directly measurable or observable. Reviewing day-of-surgery cancellations could provide resident learning opportunities in these areas.

OBJECTIVE

An automated system to facilitate anesthesiology resident review of cancelled cases was implemented on the Preoperative Evaluation Clinic (PEC) rotation at the authors' institution. This study aims to evaluate its impact on resident education.

METHODS

Residents on the PEC rotation during the 6 months preceding (n = 22) and following (n = 13) implementation in 2014 were surveyed about their experience performing cancelled case reviews in order to ascertain the effect of the intervention on their training.

RESULTS

Significant changes were reported in the number of cases reviewed by each resident (p < 0.0001), perceived importance of review (p = 0.03), and ease of review (p = 0.03) after system implementation. There was also an increase in the proportion of cancelled cases reviewed from 17.3% (34 of 196) to 95.6% (194 of 203) (p < 0.0001). Non-significant trends were seen in perceived rotation effect on ACGME competencies, including systems-based practice. Several specific improvements to our clinical practice, including the creation of standardized guidelines, arose from these case reviews.

CONCLUSION

Implementation of automated systems can improve compliance with educational goals by clarifying priorities and simplifying workflow. This system increased the number of cases reviewed by residents and the perceived importance of this review as a part of their educational experience.

Clinical exposures during internal medicine acting internship: profiling student and team experiences

BACKGROUND

The clinical learning model in medical education is driven by knowledge acquisition through direct patient-care experiences. Despite the emphasis on experiential learning, the ability of educators to quantify the clinical exposures of learners is limited.

OBJECTIVE

To utilize Veterans Affairs (VA) electronic medical record information through a data warehouse to quantify clinical exposures during an inpatient internal medicine rotation.

METHODS

We queried the VA clinical data warehouse for the patients encountered by each learner completing an acting internship rotation at the Cleveland VA Medical Center from July 2008 to November 2011. We then used discharge summary information to identify team exposures-patients seen by the learner's inpatient team who were not primarily assigned to the learner. Based on the learner and team exposures, we complied lists of past medical problems, medications prescribed, laboratory tests that resulted, radiology evaluated, and primary discharge diagnoses.

RESULTS

Primary learner and team-based clinical exposures were evaluated for a total of 128 acting internship students. The percentage of learners who had a primary exposure to a medication/lab value/imaging result/diagnosis was calculated. The percentage of learners with at least 1 primary or team-based exposure to an item was also calculated. The most common exposures in each category are presented.

CONCLUSIONS

Analysis of the clinical exposures during an inpatient rotation can augment the ability of educators to understand learners' experiences. These types of analyses could provide information to improve learner experience, implement novel curricula, and address educational gaps in clinical rotations.

Automated near-real-time clinical performance feedback for anesthesiology residents: one piece of the milestones puzzle

BACKGROUND

Anesthesiology residencies are developing trainee assessment tools to evaluate 25 milestones that map to the six core competencies. The effort will be facilitated by development of automated methods to capture, assess, and report trainee performance to program directors, the Accreditation Council for Graduate Medical Education and the trainees themselves.

METHODS

The authors leveraged a perioperative information management system to develop an automated, near-real-time performance capture and feedback tool that provides objective data on clinical performance and requires minimal administrative effort. Before development, the authors surveyed trainees about satisfaction with clinical performance feedback and about preferences for future feedback.

RESULTS

Resident performance on 24,154 completed cases has been incorporated into the authors' automated dashboard, and trainees now have access to their own performance data. Eighty percent (48 of 60) of the residents responded to the feedback survey. Overall, residents "agreed/strongly agreed" that they desire frequent updates on their clinical performance on defined quality metrics and that they desired to see how they compared with the residency as a whole. Before deployment of the new tool, they "disagreed" that they were receiving feedback in a timely manner. Survey results were used to guide the format of the feedback tool that has been implemented.

CONCLUSION

The authors demonstrate the implementation of a system that provides near-real-time feedback concerning resident performance on an extensible series of quality metrics, and which is responsive to requests arising from resident feedback about desired reporting mechanisms.

Decision support using anesthesia information management system records and accreditation council for graduate medical education case logs for resident operating room assignments

Our goal in this study was to develop decision support systems for resident operating room (OR) assignments using anesthesia information management system (AIMS) records and Accreditation Council for Graduate Medical Education (ACGME) case logs and evaluate the implementations. We developed 2 Web-based systems: an ACGME case-log visualization tool, and Residents Helping in Navigating OR Scheduling (Rhinos), an interactive system that solicits OR assignment requests from residents and creates resident profiles. Resident profiles are snapshots of the cases and procedures each resident has done and were derived from AIMS records and ACGME case logs. A Rhinos pilot was performed for 6 weeks on 2 clinical services. One hundred sixty-five requests were entered and used in OR assignment decisions by a single attending anesthesiologist. Each request consisted of a rank ordered list of up to 3 ORs. Residents had access to detailed information about these cases including surgeon and patient name, age, procedure type, and admission status. Success rates at matching resident requests were determined by comparing requests with AIMS records. Of the 165 requests, 87 first-choice matches (52.7%), 27 second-choice matches (16.4%), and 8 third-choice matches (4.8%) were made. Forty-three requests were unmatched (26.1%). Thirty-nine first-choice requests overlapped (23.6%). Full implementation followed on 8 clinical services for 8 weeks. Seven hundred fifty-four requests were reviewed by 15 attending anesthesiologists, with 339 first-choice matches (45.0%), 122 second-choice matches (16.2%), 55 third-choice matches (7.3%), and 238 unmatched (31.5%). There were 279 overlapping first-choice requests (37.0%). The overall combined match success rate was 69.4%. Separately, we developed an ACGME case-log visualization tool that allows individual resident experiences to be compared against case minimums as well as resident peer groups. We conclude that it is feasible to use ACGME case-log data in decision support systems for informing resident OR assignments. Additional analysis will be necessary to assess the educational impact of these systems.

O' surgery case log data, where art thou?

BACKGROUND

The American College of Surgeons (ACS) Case Log represents a data system that satisfies the American Board of Surgery (ABS) Maintenance of Certification (MOC) program, yet has broad data fields for surgical subspecialties. Using the ACS Case Log, we have developed a method of data capture, categorization, and reporting of acute care surgery fellows' experiences.

STUDY DESIGN

In July 2010, our acute care surgery fellowship required our fellows to log their clinical experiences into the ACS Case Log. Cases were entered similar to billable documentation rules. Keywords were entered that specified institutional services and/or resuscitation types. These data were exported in comma separated value format, deidentified, structured by Current Procedural Terminology (CPT) codes relevant to acute care surgery, and substratified by fellow and/or fellow year.

RESULTS

Fifteen report types were created consisting of operative experience by service, procedure by major category (cardiothoracic, vascular, solid organ, abdominal wall, hollow viscus, and soft tissue), total resuscitations, ultrasound, airway, ICU services, basic neurosurgery, and basic orthopaedics. Results are viewable via a secure Web application, accessible nationally, and exportable to many formats.

CONCLUSIONS

Using the ACS Case Log satisfies the ABS MOC program requirements and provides a method for monitoring and reporting acute care surgery fellow experiences. This system is flexible to accommodate the needs of surgical subspecialties and their training programs. As documentation requirements expand, efficient clinical documentation is a must for the busy surgeon. Although, our data entry and processing method has the immediate capacity for acute care surgery fellowships nationwide, multiple larger decisions regarding national case log systems should be encouraged.