Can Physician Performance Be Assessed via Simulation?

Assessment of Informed Consent and the Impact of Simulation on Anesthesia Trainees

Introduction

Over the years, the process of obtaining informed consent has evolved and now places an emphasis on the concept that patients should play a major role in medical decision making. Failure to adequately involve patients in making decisions regarding their health can lead to medicolegal consequences.

Therefore, taking informed consent is a fundamental component of anaesthesia training. Simulation, for training, is an excellent tool that is being utilised widely in the training of medical professionals. The use of simulated training for teaching the process of informed consent is an innovative initiative that can provide improved results.

Material and methods

After approval from the institutional review board, a prospective clinical study was conducted at Shaukat Khanum Memorial Cancer Hospital and Research Centre, Lahore, from August 2019 to September 2020. Sixteen anaesthesia trainees were randomly selected for the study. The study was divided into pre-interventional, interventional and post interventional phases. For data collection, a predesigned checklist was used. Data collected was analysed using SPSS version 23 (IBM Inc., Armonk, New York). The McNemar test was deployed to assess the difference between the baseline assessment and post-simulated training assessment; p-value < 0.05 was taken to be significant.

Results

Of the 16 participants, the majority were males (n= 13). A positive impact was observed in terms of improvement of the outcome of the following study components i.e., description of benefits of the procedure (p=0.01), disclosure of associated minor risks (p=0.005), disclosure of major risks (p=0.01), discussion of alternatives (p=0.001), teach back (p=0.001), documentation of patients’ verbal agreement (p=0.01), and communication skills involving utilising the process of connecting, introduction, communication, permission, response, and exit (p = 0.01).

Conclusion

Simulated training had a positive impact in improving outcomes in the following study components: description of benefits of the procedure, disclosure of associated risks, discussion of alternatives, teach back, documentation of patients’ verbal agreement, and utilisation of the process of connecting, introduction, communication, permission, responding, and exiting.

Assessment Scores of a Mock Objective Structured Clinical Examination Administered to 99 Anesthesiology Residents at 8 Institutions

BACKGROUND

Objective Structured Clinical Examinations (OSCEs) are used in a variety of high-stakes examinations. The primary goal of this study was to examine factors influencing the variability of assessment scores for mock OSCEs administered to senior anesthesiology residents.

METHODS

Using the American Board of Anesthesiology (ABA) OSCE Content Outline as a blueprint, scenarios were developed for 4 of the ABA skill types: (1) informed consent, (2) treatment options, (3) interpretation of echocardiograms, and (4) application of ultrasonography. Eight residency programs administered these 4 OSCEs to CA3 residents during a 1-day formative session. A global score and checklist items were used for scoring by faculty raters. We used a statistical framework called generalizability theory, or G-theory, to estimate the sources of variation (or facets), and to estimate the reliability (ie, reproducibility) of the OSCE performance scores. Reliability provides a metric on the consistency or reproducibility of learner performance as measured through the assessment.

RESULTS

Of the 115 total eligible senior residents, 99 participated in the OSCE because the other residents were unavailable. Overall, residents correctly performed 84% (standard deviation [SD] 16%, range 38%-100%) of the 36 total checklist items for the 4 OSCEs. On global scoring, the pass rate for the informed consent station was 71%, for treatment options was 97%, for interpretation of echocardiograms was 66%, and for application of ultrasound was 72%. The estimate of reliability expressing the reproducibility of examinee rankings equaled 0.56 (95% confidence interval [CI], 0.49-0.63), which is reasonable for normative assessments that aim to compare a resident's performance relative to other residents because over half of the observed variation in total scores is due to variation in examinee ability. Phi coefficient reliability of 0.42 (95% CI, 0.35-0.50) indicates that criterion-based judgments (eg, pass-fail status) cannot be made. Phi expresses the absolute consistency of a score and reflects how closely the assessment is likely to reproduce an examinee's final score. Overall, the greatest (14.6%) variance was due to the person by item by station interaction (3-way interaction) indicating that specific residents did well on some items but poorly on other items. The variance (11.2%) due to residency programs across case items was high suggesting moderate variability in performance from residents during the OSCEs among residency programs.

CONCLUSIONS

Since many residency programs aim to develop their own mock OSCEs, this study provides evidence that it is possible for programs to create a meaningful mock OSCE experience that is statistically reliable for separating resident performance.

Comparing Anesthesiology Residency Training Structure and Requirements in Seven Different Countries on Three Continents

Little has been published comparing the graduate medical education training structure and requirements across multiple countries. The goal of this study was to summarize and compare the characteristics of anesthesiology training programs in the USA, UK, Canada, Japan, Brazil, Denmark, and Switzerland as a way to better understand efforts to train anesthesiologists in different countries.

Two physicians trained in each of the seven countries (convenience sample) were interviewed using a semi-structured approach. The interview was facilitated by use of a predetermined questionnaire that included, for example, the duration of post-medical school training and national requirements for certain rotations, a number of cases, faculty supervision, national in-training written exams, and duty hour limits. These data were augmented by review of each country’s publicly available residency training documents as available on the internet.

Post-medical school anesthesia residency duration varied: three years (Brazil), four years (USA), five years (Canada and Switzerland), six years (Japan and Denmark) to nine years (UK), as did the number of explicitly required clinical rotations of a defined duration: zero (Denmark), one (Switzerland and UK), four (Brazil), six (Canada), and 12 (USA). Minimum case requirements exist in the USA, Japan, and Brazil, but not in the other countries. National written exams taken during training exist for all countries studied except Japan and Denmark. The countries studied increasingly aim to have competency-based education with milestone assessments. Training duty hour limits also varied including for example 37 hours/week averaged over a one month with limitations on night duties (Denmark), a weekly average of 48 hours taken over a 17 week period (UK), 50 hours/week maximum (Switzerland), 60 hours/week maximum (Brazil), and 80 hours/week averaged over four weeks (USA).

Some countries have highly structured training programs with multiple national requirements with training principally carried out at a home institution. Other countries have a more decentralized and unregulated approach with fewer (if any) specific case or rotation requirements, where the trainee creates his/her own customized training to meet broad objectives and goals. The countries studied have different national training requirements, unique duty hour rules and are at varying stages in transitioning to an outcome based model of residency.

Development and Testing of Screen-Based and Psychometric Instruments for Assessing Resident Performance in an Operating Room Simulator

Introduction. Medical simulators are used for assessing clinical skills and increasingly for testing hypotheses. We developed and tested an approach for assessing performance in anesthesia residents using screen-based simulation that ensures expert raters remain blinded to subject identity and experimental condition. Methods. Twenty anesthesia residents managed emergencies in an operating room simulator by logging actions through a custom graphical user interface. Two expert raters rated performance based on these entries using custom Global Rating Scale (GRS) and Crisis Management Checklist (CMC) instruments. Interrater reliability was measured by calculating intraclass correlation coefficients (ICC), and internal consistency of the instruments was assessed with Cronbach's alpha. Agreement between GRS and CMC was measured using Spearman rank correlation (SRC). Results. Interrater agreement (GRS: ICC = 0.825, CMC: ICC = 0.878) and internal consistency (GRS: alpha = 0.838, CMC: alpha = 0.886) were good for both instruments. Subscale analysis indicated that several instrument items can be discarded. GRS and CMC scores were highly correlated (SRC = 0.948). Conclusions. In this pilot study, we demonstrated that screen-based simulation can allow blinded assessment of performance. GRS and CMC instruments demonstrated good rater agreement and internal consistency. We plan to further test construct validity of our instruments by measuring performance in our simulator as a function of training level.

Progress in simulation education: developing an anesthesia curriculum

PURPOSE OF REVIEW

Simulation's role in anesthesia education is expanding to include more advanced skills and training for subspecialty practice. This review will provide an overview of many recent studies that expand the simulation curriculum for anesthesia education.

RECENT FINDINGS

Recent studies describe a curriculum that uses a range of simulation modalities, including part-task trainers, mannequin-based simulation, virtual reality, in-situ techniques, screen-based simulations as well as encounters with 'standardized' patients, nurses or physician colleagues. A variety of studies describe the use of task-training devises to more effectively acquire skills, such as fibre-optic intubation, ultrasound-guided regional anesthesia and transthoracic echocardiography as well as expand on a variety of teamwork skills particularly in subspecialty anesthesia practice.

SUMMARY

A curriculum is emerging that utilizes a variety of simulation modalities as part of a more comprehensive educational strategy for anesthesia specialty training.

The effect of training level on opioid utilization efficiency

STUDY OBJECTIVE

This study focuses on residents' ability to predict opioid administration requirements and if improvement is made as learners progress. Residents request opioid from the pharmacy at the start of the day based on clinical assignment. Unused and wasted opioids are returned at the end of the day. The labor and cost associated with this process are not trivial nor is the risk of excess opioid access. We examined if estimation of daily opioid use by residents increased in accuracy as progression through the program occurred.

MATERIALS AND METHODS

We conducted a retrospective analysis in the setting of the operating room examining pharmacy opioid usage. The measurements included the number of vials requested, the number of vials dispensed, and the number of vials returned. Information was stratified in our database by the level of training. The set inclusion criterion was any surgery performed in the operating rooms that required the use of opioids. The exclusion criteria included anesthesia that required a regional block, cardiac surgeries, emergent cases, and obstetric cases.

MAIN RESULTS

A total of 104 opioid requests were made that met our criteria. Comparing CA-1 and CA-3 residents and CA-2 and CA-3 residents, a statistically significant difference exists between the number of vials requested and the number returned. Our data show a statistically significant difference in efficiency as residents progress from their CA-1 year to CA-3 year.

CONCLUSION

In summary, our findings support the idea that residents are able to better predict opioid requirements for cases as they progress through training. Closely monitoring such patterns can serve a useful educational purpose and allow for identification of misuse. Improvement in cost-effective care and limiting waste while working in a complex integrated health care environment are additional benefits.