Psychometric properties of an integrated assessment of technical and communication skills

Eye tracking and audio sensors to evaluate surgeon's non-technical skills: An empirical study

Non-Technical Skills (NTS) of medical teams are currently measured using subjective and resource-intensive ratings given by experts. This study explores if objective NTS assessment approaches with eye-tracking and audio sensors can measure teamwork and communication skills in surgery. Eight surgeons participated in a simulated two-phase surgical scenario developed to assess their NTS. Sensor-based audio, eye tracking and video data were collected and analyzed along with rating from the NOTSS scale. Different levels of communication were detected by the sensor data during the two phases of the simulated surgery. Sensor data detected leadership qualities among surgeons based on speech metrics, and eye tracking offered additional evidence about gaze patterns related to NTS. This objective approach to NTS measurement captured differences in communication in greater detail as opposed to a single collective rating obtained using current assessment tools.

Wearable Systems for Unveiling Collective Intelligence in Clinical Settings

Nowadays, there is an ever-growing interest in assessing the collective intelligence (CI) of a team in a wide range of scenarios, thanks to its potential in enhancing teamwork and group performance. Recently, special attention has been devoted on the clinical setting, where breakdowns in teamwork, leadership, and communication can lead to adverse events, compromising patient safety. So far, researchers have mostly relied on surveys to study human behavior and group dynamics; however, this method is ineffective. In contrast, a promising solution to monitor behavioral and individual features that are reflective of CI is represented by wearable technologies. To date, the field of CI assessment still appears unstructured; therefore, the aim of this narrative review is to provide a detailed overview of the main group and individual parameters that can be monitored to evaluate CI in clinical settings, together with the wearables either already used to assess them or that have the potential to be applied in this scenario. The working principles, advantages, and disadvantages of each device are introduced in order to try to bring order in this field and provide a guide for future CI investigations in medical contexts.

Assessing communication skills during OSCE: need for integrated psychometric approaches

BACKGROUND

Physicians' communication skills (CS) are known to significantly affect the quality of health care. Communication skills training programs are part of most undergraduate medical curricula and are usually assessed in Objective Structured Clinical Examinations (OSCE) throughout the curriculum. The adoption of reliable measurement instruments is thus essential to evaluate such skills.

METHODS

Using Exploratory Factor Analysis (EFA), Multi-Group Confirmatory Factor Analysis (MGCFA) and Item Response Theory analysis (IRT) the current retrospective study tested the factorial validity and reliability of a four-item global rating scale developed by Hodges and McIlroy to measure CS among 296 third- and fourth-year medical students at the Faculty of Medicine in Geneva, Switzerland, during OSCEs.

RESULTS

EFA results at each station showed good reliability scores. However, measurement invariance assessments through MGCFA across different stations (i.e., same students undergoing six or three stations) and across different groups of stations (i.e., different students undergoing groups of six or three stations) were not satisfactory, failing to meet the minimum requirements to establish measurement invariance and thus possibly affecting reliable comparisons between students' communication scores across stations. IRT revealed that the four communication items provided overlapping information focusing especially on high levels of the communication spectrum.

CONCLUSIONS

Using this four-item set in its current form it may be difficult to adequately differentiate between students who are poor in CS from those who perform better. Future directions in best-practices to assess CS among medical students in the context of OSCE may thus focus on (1) training examiners so to obtain scores that are more coherent across stations; and (2) evaluating items in terms of their ability to cover a wider spectrum of medical students' CS. In this respect, IRT can prove to be very useful for the continuous evaluation of CS measurement instruments in performance-based assessments.

Interventions for improving medical students' interpersonal communication in medical consultations

BACKGROUND

Communication is a common element in all medical consultations, affecting a range of outcomes for doctors and patients. The increasing demand for medical students to be trained to communicate effectively has seen the emergence of interpersonal communication skills as core graduate competencies in medical training around the world. Medical schools have adopted a range of approaches to develop and evaluate these competencies.

OBJECTIVES

To assess the effects of interventions for medical students that aim to improve interpersonal communication in medical consultations.

SEARCH METHODS

We searched five electronic databases: Cochrane Central Register of Controlled Trials, MEDLINE, Embase, PsycINFO, and ERIC (Educational Resource Information Centre) in September 2020, with no language, date, or publication status restrictions. We also screened reference lists of relevant articles and contacted authors of included studies.

SELECTION CRITERIA

We included randomised controlled trials (RCTs), cluster-RCTs (C-RCTs), and non-randomised controlled trials (quasi-RCTs) evaluating the effectiveness of interventions delivered to students in undergraduate or graduate-entry medical programmes. We included studies of interventions aiming to improve medical students' interpersonal communication during medical consultations. Included interventions targeted communication skills associated with empathy, relationship building, gathering information, and explanation and planning, as well as specific communication tasks such as listening, appropriate structure, and question style.

DATA COLLECTION AND ANALYSIS

We used standard methodological procedures expected by Cochrane. Two review authors independently reviewed all search results, extracted data, assessed the risk of bias of included studies, and rated the quality of evidence using GRADE.

MAIN RESULTS

We found 91 publications relating to 76 separate studies (involving 10,124 students): 55 RCTs, 9 quasi-RCTs, 7 C-RCTs, and 5 quasi-C-RCTs. We performed meta-analysis according to comparison and outcome. Among both effectiveness and comparative effectiveness analyses, we separated outcomes reporting on overall communication skills, empathy, rapport or relationship building, patient perceptions/satisfaction, information gathering, and explanation and planning. Overall communication skills and empathy were further divided as examiner- or simulated patient-assessed. The overall quality of evidence ranged from moderate to very low, and there was high, unexplained heterogeneity. Overall, interventions had positive effects on most outcomes, but generally small effect sizes and evidence quality limit the conclusions that can be drawn. Communication skills interventions in comparison to usual curricula or control may improve both overall communication skills (standardised mean difference (SMD) 0.92, 95% confidence interval (CI) 0.53 to 1.31; 18 studies, 1356 participants; I² = 90%; low-quality evidence) and empathy (SMD 0.64, 95% CI 0.23 to 1.05; 6 studies, 831 participants; I² = 86%; low-quality evidence) when assessed by experts, but not by simulated patients. Students' skills in information gathering probably also improve with educational intervention (SMD 1.07, 95% CI 0.61 to 1.54; 5 studies, 405 participants; I² = 78%; moderate-quality evidence), but there may be little to no effect on students' rapport (SMD 0.18, 95% CI -0.15 to 0.51; 9 studies, 834 participants; I² = 81%; low-quality evidence), and effects on information giving skills are uncertain (very low-quality evidence). We are uncertain whether experiential interventions improve overall communication skills in comparison to didactic approaches (SMD 0.08, 95% CI -0.02 to 0.19; 4 studies, 1578 participants; I² = 4%; very low-quality evidence). Electronic learning approaches may have little to no effect on students' empathy scores (SMD -0.13, 95% CI -0.68 to 0.43; 3 studies, 421 participants; I² = 82%; low-quality evidence) or on rapport (SMD 0.02, 95% CI -0.33 to 0.38; 3 studies, 176 participants; I² = 19%; moderate-quality evidence) compared to face-to-face approaches. There may be small negative effects of electronic interventions on information giving skills (low-quality evidence), and effects on information gathering skills are uncertain (very low-quality evidence).  Personalised/specific feedback probably improves overall communication skills to a small degree in comparison to generic or no feedback (SMD 0.58, 95% CI 0.29 to 0.87; 6 studies, 502 participants; I² = 56%; moderate-quality evidence). There may be small positive effects of personalised feedback on empathy and information gathering skills (low quality), but effects on rapport are uncertain (very low quality), and we found no evidence on information giving skills. We are uncertain whether role-play with simulated patients outperforms peer role-play in improving students' overall communication skills (SMD 0.17, 95% CI -0.33 to 0.67; 4 studies, 637 participants; I² = 87%; very low-quality evidence). There may be little to no difference between effects of simulated patient and peer role-play on students' empathy (low-quality evidence) with no evidence on other outcomes for this comparison. Descriptive syntheses of results that could not be included in meta-analyses across outcomes and comparisons were mixed, as were effects of different interventions and comparisons on specific communication skills assessed by the included trials. Quality of evidence was downgraded due to methodological limitations across several risk of bias domains, high unexplained heterogeneity, and imprecision of results. In general, results remain consistent in sensitivity analysis based on risk of bias and adjustment for clustering. No adverse effects were reported.  AUTHORS' CONCLUSIONS: This review represents a substantial body of evidence from which to draw, but further research is needed to strengthen the quality of the evidence base, to consider the long-term effects of interventions on students' behaviour as they progress through training and into practice, and to assess effects of interventions on patient outcomes. Efforts to standardise assessment and evaluation of interpersonal skills will strengthen future research efforts.

Resident competencies before and after short intensive care unit rotations: a multicentre pilot observational study

PURPOSE

Residency programs need to understand the competencies developed by residents during an intensive care unit (ICU) rotation, so that curricula and assessments maximize residents' learning. The primary study objective was to evaluate the feasibility for training programs and acceptability by residents of conducting a multi-competency assessment during a four-week ICU rotation.

METHODS

We conducted a prospective, multicentre observational pilot study in three ICUs. During weeks 1 and 4 of an ICU rotation, we conducted repeated standardized assessments of non-critical care specialty residents' competencies in cognitive reasoning (script concordance test [SCT]), procedural skills (objective structured assessment of technical skills [OSATS]-global rating scale], and communication skills through a written test, two procedural simulations, and a simulated encounter with a "family member". The feasibility outcomes included program costs, the proportion of enrolled residents able to complete at least one three-station assessment during their four-week ICU rotation, and acceptability of the assessment for the trainees.

RESULTS

We enrolled 63 (69%) of 91 eligible residents, with 58 (92%) completing at least one assessment. The total cost to conduct 90 assessments was CAD 33,800. The majority of participants agreed that the assessment was fair and that it measured important clinical abilities. For the 32 residents who completed two assessments, the mean (standard deviation) cognitive reasoning and procedural skill scores increased between weeks 1 and 4 [SCT difference, 3.1 (6.5), P = 0.01; OSATS difference for bag-mask ventilation and central line insertion, 0.4 (0.5) and 0.6 (0.8), respectively; both P ≤ 0.001]. Nevertheless, the communication scores did not change significantly.

CONCLUSIONS

A monthly multi-competency assessment for specialty residents rotating in the ICU is likely feasible for most programs with appropriate resources, and generally acceptable for residents. Specialty residents' cognitive reasoning and procedural skills may improve during a four-week ICU rotation, whereas communication skills may not.

Simulation-based training in cardiology: State-of-the-art review from the French Commission of Simulation Teaching (Commission d'enseignement par simulation-COMSI) of the French Society of Cardiology

In our healthcare system, mindful of patient safety and the reduction of medical errors, simulation-based training has emerged as the cornerstone of medical education, allowing quality training in complete safety for patients. Initiated by anaesthesiologists, this teaching mode effectively allows a gradual transfer of learning, and has become an essential tool in cardiology teaching. Cardiologists are embracing simulation to master complex techniques in interventional cardiology, to manage crisis situations and unusual complications and to develop medical teamwork. Simulation methods in cardiology include high-fidelity simulators, clinical scenarios, serious games, hybrid simulation and virtual reality. Simulation involves all fields of cardiology: transoesophageal echocardiography, cardiac catheterization, coronary angioplasty and electrophysiology. Beyond purely technical issues, simulation can also enhance communication skills, by using standardized patients, and can improve the management of situations related to the announcement of serious diseases. In this review of recent literature, we present existing simulation modalities, their applications in different fields of cardiology and their advantages and limitations. Finally, we detail the growing role for simulation in the teaching of medical students following the recent legal obligation to use simulation to evaluate medical students in France.

Non-technical skills curriculum incorporating simulation-based training improves performance in colonoscopy among novice endoscopists: Randomized controlled trial

BACKGROUND AND AIMS

Non-technical skills (NTS), involving cognitive, social and interpersonal skills that complement technical skills, are important for the completion of safe and efficient procedures. We investigated the impact of a simulation-based curriculum with dedicated NTS training on novice endoscopists' performance of clinical colonoscopies.

METHODS

A single-blinded randomized controlled trial was conducted at a single center. Novice endoscopists were randomized to a control curriculum or a NTS curriculum. The control curriculum involved a didactic session, virtual reality (VR) simulator colonoscopy training, and integrated scenario practice using a VR simulator, a standardized patient, and endoscopy nurse. Feedback and training were provided by experienced endoscopists. The NTS curriculum group received similar training that included a small-group session on NTS, feedback targeting NTS, and access to a self-reflective NTS checklist. The primary outcome was performance during two clinical colonoscopies, assessed using the Joint Advisory Group Direct Observation of Procedural Skills (JAG DOPS) tool.

RESULTS

Thirty-nine participants completed the study. The NTS group (n = 21) had superior clinical performance during their first (P < 0.001) and second clinical colonoscopies (P < .0.001), compared to the control group (n = 18). The NTS group performed significantly better on the VR simulator (P < 0.05) and in the integrated scenario (P < 0.05).

CONCLUSION

Our findings demonstrate that dedicated NTS training led to improved performance of clinical colonoscopies among novices.

Developing a validity argument for a simulation-based model of entrustment in dispensing skills assessment framework

INTRODUCTION

Integrated assessment of multiple competencies at once, including entrustable professional activity (EPA) based assessment, is emerging as an effective approach to competency-based evaluation of health professionals. However, there is an absence of validated assessment frameworks in entry level pharmacy education. We aimed to develop an assessment framework and establish a validity argument, containing multiple sources of evidence, for use in the integrated assessment of pharmacy student's competency in all aspects of the supply of prescribed medicine(s).

METHODS

A two-phase prospective study was conducted. Phase 1 involved development and content validation of the Model of Entrustment in Dispensing Skills (MEDS) assessment framework using a literature review, a think-aloud study, and expert consultation. In phase 2, a pilot study was conducted with faculty and expert assessors to test the framework. Subsequent analysis involved psychometric evaluation of rating scales and usability testing.

RESULTS

Validity evidence was collected and organised across the two study phases. The MEDS framework had good evidence of content validity supported by the rigorous development and consultation process, as well as case sampling, with 88% of national practice-based competencies represented across the two simulations. Reliability coefficients were high and acceptable, supporting strong agreement across domains, students, and simulations as well as a strong correlation between the EPA and total score (spearman correlation rho 0.725, P < .001).

CONCLUSIONS

This study describes a valid and rigorous approach for the implementation and interpretation of an integrated simulation-based assessment tool for determining pharmacy student's progress towards entrustment for independent medication supply practice.

A study on evaluator factors affecting physician-patient interaction scores in clinical performance examinations: a single medical school experience

Background

This study is an analysis of evaluator factors affecting physician-patient interaction (PPI) scores in clinical performance examination (CPX). The purpose of this study was to investigate possible ways to increase the reliability of the CPX evaluation.

Methods

The six-item Yeungnam University Scale (YUS), four-item analytic global rating scale (AGRS), and one-item holistic rating scale (HRS) were used to evaluate student performance in PPI. A total of 72 fourth-year students from Yeungnam University College of Medicine in Korea participated in the evaluation with 32 faculty and 16 standardized patient (SP) raters. The study then examined the differences in scores between types of scale, raters (SP vs. faculty), faculty specialty, evaluation experience, and level of fatigue as time passes.

Results

There were significant differences between faculty and SP scores in all three scales and a significant correlation among raters’ scores. Scores given by raters on items related to their specialty were lower than those given by raters on items out of their specialty. On the YUS and AGRS, there were significant differences based on the faculty’s evaluation experience; scores by raters who had three to ten previous evaluation experiences were lower than others’ scores. There were also significant differences among SP raters on all scales. The correlation between the YUS and AGRS/HRS declined significantly according to the length of evaluation time.

Conclusion

In CPX, PPI score reliability was found to be significantly affected by the evaluator factors as well as the type of scale.

An observational study of technical and non-technical skills in advanced life support in the clinical setting

OBJECTIVE

Technical skills (TS) and non-technical skills (NTS) are the primary elements ensuring patient safety during advanced life support (ALS) and effective crisis resource management (CRM). Both skills are needed to perform high-quality ALS, though they are traditionally practiced separately. The evidence of the association between NTS and TS in high-quality ALS performance is insufficient. Hence, we aimed to evaluate the association between the skills in real-life in-hospital ALS situations.

METHODS

We video recorded real-life in-hospital ALS situations, analyzed TS and NTS demonstrated in them with an instrument measuring TS and NTS, and tested the linear association between NTS and TS using a linear mixed model.

RESULTS

Among 50 real-life in-hospital ALS situations that we recorded, 20 had adequate data for analysis. NTS and TS total scores were associated with one another (slope 0.48, P < 0.001). All NTS subcategories were associated with the TS total score (slopes ranging from 0.29 to 0.39, P < 0.001). The NTS total score and TS subcategories (chest compression quality, ventilation quality, rhythm control and defibrillation quality) were associated with one another (slopes ranging from 0.37 to 0.56, P < 0.01).

CONCLUSIONS

The resuscitation teams who demonstrated good NTS also performed the technical aspects of ALS better. The results suggest that NTS and TS have an association with one another in real-life in-hospital ALS situations. NTS performance had the most evident association with chest compression quality and rhythm control and defibrillation quality; these are considered the most crucial elements affecting outcomes of ALS. The findings of the study present novel information of what and why to emphasize in ALS training.

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov, NCT03017144.

Evaluation of Communication Skills Among Physicians: A Systematic Review of Existing Assessment Tools

OBJECTIVE

The importance of physician training in communication skills for motivating patients to adopt a healthy life-style and optimize clinical outcomes is increasingly recognized. This study inventoried and systematically reviewed the psychometric properties of, and the skills assessed by, existing assessment tools used to evaluate communication skills among physicians.

METHODS

This review was conducted in accordance with the PRISMA guidelines (PROSPERO: CRD42018091932). Four databases (PUBMED, EMBASE, PsychINFO, and SCOPUS) were searched up to December 2018, generating 3902 unique articles, which were screened by two authors. A total of 57 articles met the inclusion criteria and underwent full data extraction.

RESULTS

Forty-five different assessment tools were identified. Only 47% of the studies mentioned underlying theories or models for designing the tool. Fifteen communication skills were assessed across the tools, the five most prevalent were information giving (46%) or gathering (40%), eliciting patients' perspectives (44%), planning/goal setting (37%), and closing the session (32%). Most tools (93%) assessed communication skills using in-person role play exercises with standardized (61%) or real (32%) patients, but only 54% described the expertise of the raters who performed the evaluations. Overall, reporting of the psychometric properties of the assessment tools was poor-moderate (4.5 ± 1.3 out of 9).

CONCLUSIONS

Despite identifying several existing physician communication assessment tools, a high degree of heterogeneity between these tools, in terms of skills assessed and study quality, was observed, and most have been poorly validated. Research is needed to rigorously develop and validate accessible, convenient, "user-friendly," and easy to administer and score communication assessment tools.

Validity of the Medi-StuNTS behavioural marker system: assessing the non-technical skills of medical students during immersive simulation

Background

The Medical Students' Non-Technical Skills (Medi-StuNTS) behavioural marker system (BMS) is the first BMS to be developed specifically for medical students to facilitate training in non-technical skills (NTS) within immersive simulated acute care scenarios. In order to begin implementing the tool in practice, validity evidence must be sought. We aimed to assess the validity of the Medi-StuNTS system with reference to Messick's contemporary validity framework.

Methods

Two raters marked video-recorded performances of acute care simulation scenarios using the Medi-StuNTS system. Three groups were marked: third-year and fourth-year medical students (novices), final-year medical students (intermediates) and core medical trainees (experts). The scores were used to make assessments of relationships to the variable of clinical experience through expert-novice comparisons, inter-rater reliability, observability, exploratory factor analysis, inter-rater disagreements and differential item functioning.

Results

A significant difference was found between the three groups (p<0.005), with experts scoring significantly better than intermediates (p<0.005) and intermediates scoring significantly better than novices (p=0.001). There was a strong positive correlation between the two raters' scores (r=0.79), and an inter-rater disagreement of more than one point in less than one-fifth of cases. Across all scenarios, 99.7% of skill categories and 84% of skill elements were observable. Factor analysis demonstrated appropriate grouping of skill elements. Inconsistencies in test performance across learner groups were shown specifically in the skill categories of situation awareness and decision making and prioritisation.

Conclusion

We have demonstrated evidence for several aspects of validity of the Medi-StuNTS system when assessing medical students' NTS during immersive simulation. We can now begin to introduce this system into simulation-based education to maximise NTS training in this group.

Assessment of medical students' integrated clinical communication skills: development of a tailor-made assessment tool

BACKGROUND

Since patient-centered communication is directly connected to clinical performance, it should be integrated with medical knowledge and clinical skills. Therefore, clinical communication skills should be trained and assessed as an integral part of the student's clinical performance. We were unable to identify a tool, which helps when assessing patient-centered communication skills as an integrated component of medical history taking ('the integrated medical interview'). Therefore, we decided to design a new tailor-made assessment tool, the BOCC (BeOordeling Communicatie en Consultvoering (Dutch), Assessment of Communication and Consultation (English) to help raters assess students' integrated clinical communication skills with the emphasis on patient-centred communication combined with the correct medical content. This is a first initiative to develop such a tool, and this paper describes the first steps in this process.

METHODS

We investigated the tool in a group of third-year medical students (n = 672) interviewing simulated patients. Internal structure and internal consistency were assessed. Regression analysis was conducted to investigate the relationship between scores on the instrument and general grading. Applicability to another context was tested in a group of fourth-year medical students (n = 374).

RESULTS

PCA showed five components (Communication skills, Problem clarification, Specific History, Problem influence and Integration Skills) with various Cronbach's alpha scores. The component Problem Clarification made the strongest unique contribution to the grade prediction. Applicability was good when investigated in another context.

CONCLUSIONS

The BOCC is designed to help raters assess students' integrated communication skills. It was assessed on internal structure and internal consistency. This tool is the first step in the assessment of the integrated medical interview and a basis for further investigation to reform it into a true measurement instrument on clinical communication skills.

Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy

BACKGROUND

Endoscopy has traditionally been taught with novices practicing on real patients under the supervision of experienced endoscopists. Recently, the growing awareness of the need for patient safety has brought simulation training to the forefront. Simulation training can provide trainees with the chance to practice their skills in a learner-centred, risk-free environment. It is important to ensure that skills gained through simulation positively transfer to the clinical environment. This updated review was performed to evaluate the effectiveness of virtual reality (VR) simulation training in gastrointestinal endoscopy.

OBJECTIVES

To determine whether virtual reality simulation training can supplement and/or replace early conventional endoscopy training (apprenticeship model) in diagnostic oesophagogastroduodenoscopy, colonoscopy, and/or sigmoidoscopy for health professions trainees with limited or no prior endoscopic experience.

SEARCH METHODS

We searched the following health professions, educational, and computer databases until 12 July 2017: the Cochrane Central Register of Controlled Trials, Ovid MEDLINE, Ovid Embase, Scopus, Web of Science, BIOSIS Previews, CINAHL, AMED, ERIC, Education Full Text, CBCA Education, ACM Digital Library, IEEE Xplore, Abstracts in New Technology and Engineering, Computer and Information Systems Abstracts, and ProQuest Dissertations and Theses Global. We also searched the grey literature until November 2017.

SELECTION CRITERIA

We included randomised and quasi-randomised clinical trials comparing VR endoscopy simulation training versus any other method of endoscopy training with outcomes measured on humans in the clinical setting, including conventional patient-based training, training using another form of endoscopy simulation, or no training. We also included trials comparing two different methods of VR training.

DATA COLLECTION AND ANALYSIS

Two review authors independently assessed the eligibility and methodological quality of trials, and extracted data on the trial characteristics and outcomes. We pooled data for meta-analysis where participant groups were similar, studies assessed the same intervention and comparator, and had similar definitions of outcome measures. We calculated risk ratio for dichotomous outcomes with 95% confidence intervals (CI). We calculated mean difference (MD) and standardised mean difference (SMD) with 95% CI for continuous outcomes when studies reported the same or different outcome measures, respectively. We used GRADE to rate the quality of the evidence.

MAIN RESULTS

We included 18 trials (421 participants; 3817 endoscopic procedures). We judged three trials as at low risk of bias. Ten trials compared VR training with no training, five trials with conventional endoscopy training, one trial with another form of endoscopy simulation training, and two trials compared two different methods of VR training. Due to substantial clinical and methodological heterogeneity across our four comparisons, we did not perform a meta-analysis for several outcomes. We rated the quality of evidence as moderate, low, or very low due to risk of bias, imprecision, and heterogeneity.Virtual reality endoscopy simulation training versus no training: There was insufficient evidence to determine the effect on composite score of competency (MD 3.10, 95% CI -0.16 to 6.36; 1 trial, 24 procedures; low-quality evidence). Composite score of competency was based on 5-point Likert scales assessing seven domains: atraumatic technique, colonoscope advancement, use of instrument controls, flow of procedure, use of assistants, knowledge of specific procedure, and overall performance. Scoring range was from 7 to 35, a higher score representing a higher level of competence. Virtual reality training compared to no training likely provides participants with some benefit, as measured by independent procedure completion (RR 1.62, 95% CI 1.15 to 2.26; 6 trials, 815 procedures; moderate-quality evidence). We evaluated overall rating of performance (MD 0.45, 95% CI 0.15 to 0.75; 1 trial, 18 procedures), visualisation of mucosa (MD 0.60, 95% CI 0.20 to 1.00; 1 trial, 55 procedures), performance time (MD -0.20 minutes, 95% CI -0.71 to 0.30; 2 trials, 29 procedures), and patient discomfort (SMD -0.16, 95% CI -0.68 to 0.35; 2 trials, 145 procedures), all with very low-quality evidence. No trials reported procedure-related complications or critical flaws (e.g. bleeding, luminal perforation) (3 trials, 550 procedures; moderate-quality evidence).Virtual reality endoscopy simulation training versus conventional patient-based training: One trial reported composite score of competency but did not provide sufficient data for quantitative analysis. Virtual reality training compared to conventional patient-based training resulted in fewer independent procedure completions (RR 0.45, 95% CI 0.27 to 0.74; 2 trials, 174 procedures; low-quality evidence). We evaluated performance time (SMD 0.12, 95% CI -0.55 to 0.80; 2 trials, 34 procedures), overall rating of performance (MD -0.90, 95% CI -4.40 to 2.60; 1 trial, 16 procedures), and visualisation of mucosa (MD 0.0, 95% CI -6.02 to 6.02; 1 trial, 18 procedures), all with very low-quality evidence. Virtual reality training in combination with conventional training appears to be advantageous over VR training alone. No trials reported any procedure-related complications or critical flaws (3 trials, 72 procedures; very low-quality evidence).Virtual reality endoscopy simulation training versus another form of endoscopy simulation: Based on one study, there were no differences between groups with respect to composite score of competency, performance time, and visualisation of mucosa. Virtual reality training in combination with another form of endoscopy simulation training did not appear to confer any benefit compared to VR training alone.Two methods of virtual reality training: Based on one study, a structured VR simulation-based training curriculum compared to self regulated learning on a VR simulator appears to provide benefit with respect to a composite score evaluating competency. Based on another study, a progressive-learning curriculum that sequentially increases task difficulty provides benefit with respect to a composite score of competency over the structured VR training curriculum.

AUTHORS' CONCLUSIONS

VR simulation-based training can be used to supplement early conventional endoscopy training for health professions trainees with limited or no prior endoscopic experience. However, we found insufficient evidence to advise for or against the use of VR simulation-based training as a replacement for early conventional endoscopy training. The quality of the current evidence was low due to inadequate randomisation, allocation concealment, and/or blinding of outcome assessment in several trials. Further trials are needed that are at low risk of bias, utilise outcome measures with strong evidence of validity and reliability, and examine the optimal nature and duration of training.

Progressive learning in endoscopy simulation training improves clinical performance: a blinded randomized trial

BACKGROUND AND AIMS

A structured comprehensive curriculum (SCC) that uses simulation-based training (SBT) can improve clinical colonoscopy performance. This curriculum may be enhanced through the application of progressive learning, a training strategy centered on incrementally challenging learners. We aimed to determine whether a progressive learning-based curriculum (PLC) would lead to superior clinical performance compared with an SCC.

METHODS

This was a single-blinded randomized controlled trial conducted at a single academic center. Thirty-seven novice endoscopists were recruited and randomized to either a PLC (n = 18) or to an SCC (n = 19). The PLC comprised 6 hours of SBT, which progressed in complexity and difficulty. The SCC included 6 hours of SBT, with cases of random order of difficulty. Both groups received expert feedback and 4 hours of didactic teaching. Participants were assessed at baseline, immediately after training, and 4 to 6 weeks after training. The primary outcome was participants' performance during their first 2 clinical colonoscopies, as assessed by using the Joint Advisory Group Direct Observation of Procedural Skills assessment tool (JAG DOPS). Secondary outcomes were differences in endoscopic knowledge, technical and communication skills, and global performance in the simulated setting.

RESULTS

The PLC group outperformed the SCC group during first and second clinical colonoscopies, measured by JAG DOPS (P < .001). Additionally, the PLC group had superior technical and communication skills and global performance in the simulated setting (P < .05). There were no differences between groups in endoscopic knowledge (P > .05).

CONCLUSIONS

Our findings demonstrate the superiority of a PLC for endoscopic simulation, compared with an SCC. Challenging trainees progressively is a simple, theory-based approach to simulation whereby the performance of clinical colonoscopies can be improved. (Clinical trial registration number: NCT02000180.).

Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial

Background

Non-technical skills (NTS), such as communication and professionalism, contribute to the safe and effective completion of procedures. NTS training has previously been shown to improve surgical performance. Moreover, increases in NTS have been associated with improved clinical endoscopic performance. Despite this evidence, NTS training has not been tested as an intervention in endoscopy.

Objective

The aim of this study is to evaluate the effectiveness of a simulation-based training (SBT) curriculum of NTS on novice endoscopists’ performance of clinical colonoscopy.

Methods

Novice endoscopists were randomized to 2 groups. The control group received 4 hours of interactive didactic sessions on colonoscopy theory and 6 hours of SBT. Hours 5 and 6 of the SBT were integrated scenarios, wherein participants interacted with a standardized patient and nurse, while performing a colonoscopy on the virtual reality (VR) simulator. The NTS (intervention) group received the same teaching sessions but the last hour was focused on NTS teaching. The NTS group also reviewed a checklist of tasks relevant to NTS concepts prior to each integrated scenario case and was provided with dedicated feedback on their NTS performance during the integrated scenario practice. All participants were assessed at baseline, immediately after training, and 4 to 6 weeks post-training. The primary outcome measure is colonoscopy-specific performance in the clinical setting.

Results

In total, 42 novice endoscopists completed the study. Data collection and analysis is ongoing. We anticipate completion of all assessments by August 2017. Data analysis, manuscript writing, and subsequent submission for publication is expected to be completed by December 2017.

Conclusions

Results from this study may inform the implementation of NTS training into postgraduate gastrointestinal curricula. NTS curricula may improve attitudes towards patient safety and self-reflection among trainees. Moreover, enhanced NTS may lead to superior clinical performance and outcomes in colonoscopy.

Trial Registration

Clinicaltrial.gov NCT02877420; https://www.clinicaltrials.gov/ct2/show/NCT02877420 (Archived by WebCite at http://www.webcitation.org/6rw94ubXX NCT02877420)

The effect on reliability and sensitivity to level of training of combining analytic and holistic rating scales for assessing communication skills in an internal medicine resident OSCE

OBJECTIVE

Although previous research has compared checklists to rating scales for assessing communication, the purpose of this study was to compare the effect on reliability and sensitivity to level of training of an analytic, a holistic, and a combined analytic-holistic rating scale in assessing communication skills.

METHODS

The University of Alberta Internal Medicine Residency runs OSCEs for postgraduate year (PGY) 1 and 2 residents and another for PGY-4 residents. Communication stations were scored with an analytic scale (empathy, non-verbal skills, verbal skills, and coherence subscales) and a holistic scale. Authors analyzed reliability of individual and combined scales using generalizability theory and evaluated each scale's sensitivity to level of training.

RESULTS

For analytic, holistic, and combined scales, 12, 12, and 11 stations respectively yielded a Phi of 0.8 for the PGY-1,2 cohort, and 16, 16, and 14 stations yielded a Phi of 0.8 for the PGY-4 cohort. PGY-4 residents scored higher on the combined scale, the analytic rating scale, and the non-verbal and coherence subscales.

CONCLUSION

A combined analytic-holistic rating scale increased score reliability and was sensitive to level of training.

PRACTICE IMPLICATIONS

Given increased validity evidence, OSCE developers should consider combining analytic and holistic scales when assessing communication skills.

Surgical education and adult learning: Integrating theory into practice

Surgical education continues to evolve from the master-apprentice model. Newer methods of the process need to be used to manage the dual challenges of educating while providing safe surgical care. This requires integrating adult learning concepts into delivery of practical training and education in busy clinical environments. A narrative review aimed at outlining and integrating adult learning and surgical education theory was undertaken. Additionally, this information was used to relate the practical delivery of surgical training and education in day-to-day surgical practice. Concepts were sourced from reference material. Additional material was found using a PubMed search of the words: 'surgical education theory' and 'adult learning theory medical'. This yielded 1351 abstracts, of which 43 articles with a focus on key concepts in adult education theory were used. Key papers were used to formulate structure and additional cross-referenced papers were included where appropriate. Current concepts within adult learning have a lot to offer when considering how to better deliver surgical education and training. Better integration of adult learning theory can be fruitful. Individual teaching surgical units need to rethink their paradigms and consider how each individual can contribute to the education experience. Up skilling courses for trainers can do much to improve the delivery of surgical education. Understanding adult learning concepts and integrating these into day-to-day teaching can be valuable.

Validity and Feasibility Evidence of Objective Structured Clinical Examination to Assess Competencies of Pediatric Critical Care Trainees

OBJECTIVE

The purpose of this study was to provide validity and feasibility evidence for the use of an objective structured clinical examination in the assessment of pediatric critical care medicine trainees.

DESIGN

This was a validation study. Validity evidence was based on Messick's framework.

SETTING

A tertiary, university-affiliated academic center.

SUBJECTS

Seventeen pediatric critical care medicine fellows were recruited in 2012 and 2013 academic year.

INTERVENTIONS

None. All subjects completed an objective structured clinical examination assessment.

MEASUREMENTS AND MAIN RESULTS

Seventeen trainees were assessed. Simulation scenarios were developed for content validity by pediatric critical care medicine and education experts using CanMEDS competencies. Scenarios were piloted before the study. Each scenario was evaluated by two interprofessional raters. Inter-rater agreement, measured using intraclass correlations, was 0.91 (SE = 0.09) across stations. Generalizability theory was used to evaluate internal structure and reliability. Reliability was moderate (G-coefficient = 0.67, Φ-coefficient = 0.52). The greatest source of variability was from participant by station variance (40.6%). Pearson correlation coefficients were used to evaluate the relationship of objective structured clinical examination with each traditional assessment instruments: multisource feedback, in-training evaluation report, short-answer questions, and Multidisciplinary Critical Care Knowledge Assessment Program. Performance on the objective structured clinical examination correlated with performance on the Multidisciplinary Critical Care Knowledge Assessment Program (r = 0.52; p = 0.032) and multisource feedback (r = 0.59; p = 0.017), but not with the overall performance on the in-training evaluation report (r = 0.37; p = 0.143) or short-answer questions (r = 0.08; p = 0.767). Consequences were not assessed.

CONCLUSION

Validity and feasibility evidence in this study indicate that the use of the objective structured clinical examination scores can be a valid way to assess CanMEDS competencies required for independent practice in pediatric critical care medicine.

Consequences Validity Evidence: Evaluating the Impact of Educational Assessments

Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation.Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor.Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

Tools to Assess Behavioral and Social Science Competencies in Medical Education: A Systematic Review

PURPOSE

Behavioral and social science (BSS) competencies are needed to provide quality health care, but psychometrically validated measures to assess these competencies are difficult to find. Moreover, they have not been mapped to existing frameworks, like those from the Liaison Committee on Medical Education (LCME) and Accreditation Council for Graduate Medical Education (ACGME). This systematic review aimed to identify and evaluate the quality of assessment tools used to measure BSS competencies.

METHOD

The authors searched the literature published between January 2002 and March 2014 for articles reporting psychometric or other validity/reliability testing, using OVID, CINAHL, PubMed, ERIC, Research and Development Resource Base, SOCIOFILE, and PsycINFO. They reviewed 5,104 potentially relevant titles and abstracts. To guide their review, they mapped BSS competencies to existing LCME and ACGME frameworks. The final included articles fell into three categories: instrument development, which were of the highest quality; educational research, which were of the second highest quality; and curriculum evaluation, which were of lower quality.

RESULTS

Of the 114 included articles, 33 (29%) yielded strong evidence supporting tools to assess communication skills, cultural competence, empathy/compassion, behavioral health counseling, professionalism, and teamwork. Sixty-two (54%) articles yielded moderate evidence and 19 (17%) weak evidence. Articles mapped to all LCME standards and ACGME core competencies; the most common was communication skills.

CONCLUSIONS

These findings serve as a valuable resource for medical educators and researchers. More rigorous measurement validation and testing and more robust study designs are needed to understand how educational strategies contribute to BSS competency development.

Exploring the relationship between anaesthesiologists' non-technical and technical skills

BACKGROUND

A combination of non-technical skills (NTS) and technical skills (TS) is crucial for anaesthetic patient management. However, a deeper understanding of the relationship between these two skills remains to be explored. We investigated the characteristics of trainee anaesthesiologists' NTS and TS in a simulated unexpected difficult airway management scenario.

METHODS

A mixed-method approach was used to explore the relationship between NTS and TS in 25 videos of 2nd year trainee anaesthesiologists managing a simulated difficult airway scenario. The videos were assessed using the customised version of the Anaesthetists' Non-Technical Skills System, ANTSdk, and an adapted TS checklist for calculating the correlation between NTS and TS. Written descriptions of the observed NTS were analysed using directed content analysis.

RESULTS

The correlation between the NTS and the TS ratings was 0.106 (two-tailed significance of 0.613). Inter-rater reliability was substantial. Themes characterising good NTS included a systematic approach, planning and communicating decisions as well as responding to the evolving situation. A list of desirable, concrete NTS for the specific airway management situation was generated.

CONCLUSION

This study illustrates that anaesthesiologist trainees' NTS and TS were not correlated in this setting, but rather intertwined and how the interplay of NTS and TS can impact patient management. Themes describing the characteristics of NTS and a list of desirable, concrete NTS were developed to aid the understanding, training and use of NTS.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2%) and non-technical (83.2 vs. 75.1%) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

The effectiveness of varied levels of simulation fidelity on integrated performance of technical skills in midwifery students--a randomised intervention trial

BACKGROUND

Simulation as a pedagogical approach is used in health professional education to address the need to safely develop effective clinical skills prior to undertaking clinical practice, in complex healthcare environments. Evidence for the use of simulation in midwifery is largely anecdotal, and research evaluating the effectiveness of different levels of simulation fidelity is lacking.

OBJECTIVES

To evaluate the effectiveness of varying levels of fidelity on simulated learning experiences and identify which best contributes to integrated and global clinical skills development in midwifery students.

DESIGN

Randomised three arm intervention trial.

PARTICIPANTS

Midwifery students who had yet to receive theoretical instruction in the performance of the clinical skill of vaginal examination.

METHODS

Midwifery students (n=69) received theoretical instruction in the performance of vaginal examination following random allocation into one of three intervention arms. Participants were recorded performing the procedure using low fidelity (part task trainer only), medium fidelity (part task trainer and life sized poster of a pregnant woman) or progressive fidelity (part task trainer and a simulated standardised patient). Senior midwifery students were recruited to act in the role of standardised patients.

RESULTS

There was a statistically significant difference in the mean total Global Rating Scale score between at least two of the three groups (p=0.009). The progressive fidelity group revealed as different from both the low fidelity group (p=0.010) and medium fidelity group (p=0.048). There was a statistically significant difference in the mean total Integrated Procedural Performance Instrument score between at least two of the three groups (p=0.012). The progressive fidelity group revealed as different from both the low fidelity group (p=0.026) and medium fidelity group (p=0.026).

CONCLUSIONS

Progressive and medium fidelity simulation yields better outcomes than low fidelity simulation and where resources are constrained medium fidelity equipment, such as a life sized poster can produce effective learning experiences for midwifery students.

A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment

CONTEXT

The relative advantages and disadvantages of checklists and global rating scales (GRSs) have long been debated. To compare the merits of these scale types, we conducted a systematic review of the validity evidence for checklists and GRSs in the context of simulation-based assessment of health professionals.

METHODS

We conducted a systematic review of multiple databases including MEDLINE, EMBASE and Scopus to February 2013. We selected studies that used both a GRS and checklist in the simulation-based assessment of health professionals. Reviewers working in duplicate evaluated five domains of validity evidence, including correlation between scales and reliability. We collected information about raters, instrument characteristics, assessment context, and task. We pooled reliability and correlation coefficients using random-effects meta-analysis.

RESULTS

We found 45 studies that used a checklist and GRS in simulation-based assessment. All studies included physicians or physicians in training; one study also included nurse anaesthetists. Topics of assessment included open and laparoscopic surgery (n = 22), endoscopy (n = 8), resuscitation (n = 7) and anaesthesiology (n = 4). The pooled GRS-checklist correlation was 0.76 (95% confidence interval [CI] 0.69-0.81, n = 16 studies). Inter-rater reliability was similar between scales (GRS 0.78, 95% CI 0.71-0.83, n = 23; checklist 0.81, 95% CI 0.75-0.85, n = 21), whereas GRS inter-item reliabilities (0.92, 95% CI 0.84-0.95, n = 6) and inter-station reliabilities (0.80, 95% CI 0.73-0.85, n = 10) were higher than those for checklists (0.66, 95% CI 0-0.84, n = 4 and 0.69, 95% CI 0.56-0.77, n = 10, respectively). Content evidence for GRSs usually referenced previously reported instruments (n = 33), whereas content evidence for checklists usually described expert consensus (n = 26). Checklists and GRSs usually had similar evidence for relations to other variables.

CONCLUSIONS

Checklist inter-rater reliability and trainee discrimination were more favourable than suggested in earlier work, but each task requires a separate checklist. Compared with the checklist, the GRS has higher average inter-item and inter-station reliability, can be used across multiple tasks, and may better capture nuanced elements of expertise.

What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment

Ongoing transformations in health professions education underscore the need for valid and reliable assessment. The current standard for assessment validation requires evidence from five sources: content, response process, internal structure, relations with other variables, and consequences. However, researchers remain uncertain regarding the types of data that contribute to each evidence source. We sought to enumerate the validity evidence sources and supporting data elements for assessments using technology-enhanced simulation. We conducted a systematic literature search including MEDLINE, ERIC, and Scopus through May 2011. We included original research that evaluated the validity of simulation-based assessment scores using two or more evidence sources. Working in duplicate, we abstracted information on the prevalence of each evidence source and the underlying data elements. Among 217 eligible studies only six (3 %) referenced the five-source framework, and 51 (24 %) made no reference to any validity framework. The most common evidence sources and data elements were: relations with other variables (94 % of studies; reported most often as variation in simulator scores across training levels), internal structure (76 %; supported by reliability data or item analysis), and content (63 %; reported as expert panels or modification of existing instruments). Evidence of response process and consequences were each present in <10 % of studies. We conclude that relations with training level appear to be overrepresented in this field, while evidence of consequences and response process are infrequently reported. Validation science will be improved as educators use established frameworks to collect and interpret evidence from the full spectrum of possible sources and elements.

Communication skills training in surgical residency: a needs assessment and metacognition analysis of a difficult conversation objective structured clinical examination

BACKGROUND

The objective structured clinical examination (OSCE) can be used to evaluate the Accreditation Council for Graduate Medical Education Core Competencies of Professionalism and Interpersonal and Communication Skills. The aim of this study was to describe general surgery resident performance on a "difficult conversation" OSCE.

METHODS

In this prospective study, junior and senior residents participated in a 2-station OSCE. Junior stations involved discussing operative risks and benefits and breaking bad news. Senior stations involved discussing goals of care and discussing transition to comfort measures only status. Residents completed post-OSCE checklist and Likert-based self-evaluations of experience, comfort, and confidence. Trained standardized patients (SPs) evaluated residents using communication skill-based checklists and Likert-based assessments. Pearson correlation coefficients were determined between self-assessment and SP assessment. Mann-Whitney U tests were conducted between junior and senior resident variables, using α = 0.05.

RESULTS

There were 27 junior residents (age 28.1 ± 1.9 years [29.6% female]) and 27 senior residents (age 32.1 ± 2.5 years [26.9% female]). The correlation of self-assessment and SP assessment of overall communication skills by junior residents was -0.32 on the risks and benefits case and 0.07 on the breaking bad news case. The correlation of self-assessment and SP assessment of overall communication skills by senior residents was 0.30 on the goals of care case and 0.26 on the comfort measures only case. SP assessments showed that junior residents had higher overall communication skills than senior residents (p = 0.03). Senior residents perceived that having difficult conversations was more level appropriate (p < 0.001), and they were less nervous having difficult conversations (p < 0.01) than junior residents.

CONCLUSIONS

We found that residents perform difficult conversations well, that subjective and objective skills are correlated, and that skills-based training is needed across all residency levels. This well-received method may be used to observe, document, and provide resident feedback for these important skills.

A qualitative study on trainees' and supervisors' perceptions of assessment for learning in postgraduate medical education

INTRODUCTION

Recent changes in postgraduate medical training curricula usually encompass a shift towards more formative assessment, or assessment for learning. However, though theoretically well suited to postgraduate training, evidence is emerging that engaging in formative assessment in daily clinical practice is complex.

AIM

We aimed to explore trainees' and supervisors' perceptions of what factors determine active engagement in formative assessment.

METHODS

Focus group study with postgraduate trainees and supervisors in obstetrics and gynaecology.

RESULTS

Three higher order themes emerged: individual perspectives on feedback, supportiveness of the learning environment and the credibility of feedback and/or feedback giver.

CONCLUSION

Engaging in formative assessment with a genuine impact on learning is complex and quite a challenge to both trainees and supervisors. Individual perspectives on feedback, a supportive learning environment and credibility of feedback are all important in this process. Every one of these should be taken into account when the utility of formative assessment in postgraduate medical training is evaluated.

A protocol for evaluating progressive levels of simulation fidelity in the development of technical skills, integrated performance and woman centred clinical assessment skills in undergraduate midwifery students

BACKGROUND

Simulation as a pedagogical approach has been used in health professional education to address the need to safely develop effective clinical skills prior to undertaking clinical practice. However, evidence for the use of simulation in midwifery is largely anecdotal, and research evaluating the effectiveness of different levels of simulation fidelity are lacking.Woman centred care is a core premise of the midwifery profession and describes the behaviours of an individual midwife who demonstrates safe and effective care of the individual woman. Woman centred care occurs when the midwife modifies the care to ensure the needs of each individual woman are respected and addressed. However, a review of the literature demonstrates an absence of a valid and reliable tool to measure the development of woman centred care behaviours. This study aims to determine which level of fidelity in simulated learning experiences provides the most effective learning outcomes in the development of woman centred clinical assessment behaviors and skills in student midwives.

METHODS/DESIGN

Three-arm, randomised, intervention trial.In this research we plan to:a) trial three levels of simulation fidelity - low, medium and progressive, on student midwives performing the procedure of vaginal examination;b) measure clinical assessment skills using the Global Rating Scale (GRS) and Integrated Procedural Performance Instrument (IPPI); andc) pilot the newly developed Woman Centred Care Scale (WCCS) to measure clinical behaviors related to Woman-Centredness.

DISCUSSION

This project aims to enhance knowledge in relation to the appropriate levels of fidelity in simulation that yield the best educational outcomes for the development of woman centred clinical assessment in student midwives. The outcomes of this project may contribute to improved woman centred clinical assessment for student midwives, and more broadly influence decision making regarding education resource allocation for maternity simulation.

Teaching and assessing procedural skills: a qualitative study

BACKGROUND

Graduating Internal Medicine residents must possess sufficient skills to perform a variety of medical procedures. Little is known about resident experiences of acquiring procedural skills proficiency, of practicing these techniques, or of being assessed on their proficiency. The purpose of this study was to qualitatively investigate resident 1) experiences of the acquisition of procedural skills and 2) perceptions of procedural skills assessment methods available to them.

METHODS

Focus groups were conducted in the weeks following an assessment of procedural skills incorporated into an objective structured clinical examination (OSCE). Using fundamental qualitative description, emergent themes were identified and analyzed.

RESULTS

Residents perceived procedural skills assessment on the OSCE as a useful formative tool for direct observation and immediate feedback. This positive reaction was regularly expressed in conjunction with a frustration with available assessment systems. Participants reported that proficiency was acquired through resident directed learning with no formal mechanism to ensure acquisition or maintenance of skills.

CONCLUSIONS

The acquisition and assessment of procedural skills in Internal Medicine programs should move toward a more structured system of teaching, deliberate practice and objective assessment. We propose that directed, self-guided learning might meet these needs.

Hybrid simulation for knee arthrocentesis: improving fidelity in procedures training

BACKGROUND

Procedures form a core competency for internists, yet many do not master these skills during residency. Simulation can help fill this gap, but many curricula focus on technical skills, and overlook communication skills necessary to perform procedures proficiently. Hybrid simulation (HS) is a novel way to teach and assess procedural skills in an integrated, contextually-based way.

AIM

To create a HS model for teaching arthrocentesis to internal medicine residents.

SETTING

Internal medicine residency program at the University of Toronto.

PARTICIPANTS

Twenty four second-year internal medicine residents.

PROGRAM DESCRIPTION

Residents were introduced to HS, given practice time with feedback from standardized patients (SPs) and faculty, and assessed individually using a different scenario and SP. Physicians scored overall performance using a 6-point procedural skills measure, and both physicians and SPs scored communication using a 5-point communication skills measure.

PROGRAM EVALUATION

Realism was highly rated by residents (4.13/5.00), SPs (4.00) and physicians (4.33), and was perceived to enhance learning. Residents' procedural skills were rated as 4.21/6.00 (3.00 - 5.00; ICC = 0.77, [0.53 - 0.92]), comparable to an experienced post-graduate year (PGY) 2-3; and all but one resident was considered competent.

DISCUSSION

HS facilitates simultaneous acquisition of technical and communication skills. Future research should examine whether HS improves transfer of skills to the clinical setting.

Development and Validation of Tools for Evaluation of Orthosis Fabrication

This study is the first phase of research aimed at developing new educational approaches to enhance occupational therapy students’ orthosis fabrication skills. Before the effectiveness of training can be determined, a method for evaluating performance must be established. Using the Delphi method, we developed a global rating scale and checklist for evaluating technical competence when fabricating metacarpophalangeal (MCP) joint–stabilizing orthoses. To determine the reliability and validity of these tools, three evaluators used them to assess and score orthotic fabrication performance by experienced and student occupational therapists. The results suggest that these measurement tools are valid and reliable indicators of the technical skills involved in fabricating an MCP joint–stabilizing orthosis. Future studies should focus on building on these tools to evaluate communication skills, technical skills for making other types of orthoses, and effectiveness of training programs.

Evaluating pain education programs: an integrated approach

Evaluation of educational programs and assessment of learning are essential to maintain high-standard health science education, which includes pain education. Current models of program evaluations applied to the education of the health professions, such as the Kirkpatrick model, are mainly outcome based. More recently, efforts have been made to examine other process-based models such as the Context Input Process Product model. The present article proposes an approach that integrates both outcome- and process-based models with models of clinical performance assessment to provide a deeper understanding of a program function. Because assessment instruments are a critical part of program evaluation, it is suggested that standardization and rigour should be used in their selection, development and adaptation. The present article suggests an alternative to currently used models in pain education evaluation.

The impact of nontechnical skills on technical performance in surgery: a systematic review

BACKGROUND

Failures in nontechnical and teamwork skills frequently lie at the heart of harm and near-misses in the operating room (OR). The purpose of this systematic review was to assess the impact of nontechnical skills on technical performance in surgery.

STUDY DESIGN

MEDLINE, EMBASE, PsycINFO databases were searched, and 2,041 articles were identified. After limits were applied, 341 articles were retrieved for evaluation. Of these, 28 articles were accepted for this review. Data were extracted from the articles regarding sample population, study design and setting, measures of nontechnical skills and technical performance, study findings, and limitations.

RESULTS

Of the 28 articles that met inclusion criteria, 21 articles assessed the impact of surgeons' nontechnical skills on their technical performance. The evidence suggests that receiving feedback and effectively coping with stressful events in the OR has a beneficial impact on certain aspects of technical performance. Conversely, increased levels of fatigue are associated with detriments to surgical skill. One article assessed the impact of anesthesiologists' nontechnical skills on anesthetic technical performance, finding a strong positive correlation between the 2 skill sets. Finally, 6 articles assessed the impact of multiple nontechnical skills of the entire OR team on surgical performance. A strong relationship between teamwork failure and technical error was empirically demonstrated in these studies.

CONCLUSIONS

Evidence suggests that certain nontechnical aspects of performance can enhance or, if lacking, contribute to deterioration of surgeons' technical performance. The precise extent of this effect remains to be elucidated.

Short simulation training improves objective skills in established advanced practitioners managing emergencies on the ward and surgical intensive care unit

BACKGROUND

Several studies evaluating simulation training in intensive care unit (ICU) physicians have demonstrated improvement in leadership and management skills. No study to date has evaluated whether such training is useful in established ICU advanced practitioners (APs). We hypothesized that human patient simulator-based training would improve surgical ICU APs' skills at managing medical crises.

METHODS

After institutional review board approval, 12 APs completed ½ day of simulation training on the SimMan, Laerdal system. Each subject participated in five scenarios, first as team leader (pretraining scenario), then as observer for three scenarios, and finally, again as team leader (posttraining). Faculty teaching accompanied each scenario and preceded a debriefing session with video replay. Three experts scored emergency care skills (Airway-Breathing-Circulation [ABCs] sequence, recognition of shock, pneumothorax, etc.) and teamwork leadership/interpersonal skills. A multiple choice question examination and training effectiveness questionnaire were completed before and after training. Fellows underwent the same curriculum and served to validate the study. Pre- and postscores were compared using the Wilcoxon signed rank test with two-tailed significance of 0.05.

RESULTS

Improvement was seen in participants' scores combining all parameters (73% ± 13% vs. 80% ± 11%, p = 0.018). AP leadership/interpersonal skills (+12%), multiple choice question examination (+4%), and training effectiveness questionnaire (+6%) scores improved significantly (p < 0.05). Fellows teamwork leadership/interpersonal skills scores were higher than APs (p < 0.001) but training brought AP scores to fellow levels. Interrater reliability was high (r = 0.77, 95% confidence interval 0.71-0.82; p < 0.001).

CONCLUSIONS

Human patient simulator training in established surgical ICU APs improves leadership, teamwork, and self-confidence skills in managing medical emergencies. Such a validated curriculum may be useful as an AP continuing education resource.

Medical simulation in respiratory and critical care medicine

Simulation-based medical education has gained tremendous popularity over the past two decades. Driven by the patient safety movement, changes in the educational opportunities available to trainees and the rapidly evolving capabilities of computer technology, simulation-based medical education is now being used across the continuum of medical education. This review provides the reader with a perspective on simulation specific to respiratory and critical care medicine, including an overview of historical and modern simulation modalities and the current evidence supporting their use.

Coordinating progressive levels of simulation fidelity to maximize educational benefit

PURPOSE

To evaluate the effectiveness of a novel, simulation-based educational model rooted in scaffolding theory that capitalizes on a systematic progressive sequence of simulators that increase in realism (i.e., fidelity) and information content.

METHOD

Forty-five medical students were randomly assigned to practice intravenous catheterization using high-fidelity training, low-fidelity training, or progressive training from low to mid to high fidelity. One week later, participants completed a transfer test on a standardized patient simulation. Blinded expert raters assessed participants' global clinical performance, communication, procedure documentation, and technical skills on the transfer test. Participants' management of the resources available during practice was also recorded. Data were analyzed using multivariate analysis of variance. The study was conducted in fall 2008 at the University of Toronto.

RESULTS

The high-fidelity group scored higher (P < .05) than the low-fidelity group on all measures except procedure documentation. The progressive group scored higher (P < .05) than other groups for documentation and global clinical performance and was equivalent to the high-fidelity group for communication and technical skills. Total practice time was greatest for the progressive group; however, this group required little practice time on the resource-intensive high-fidelity simulator.

CONCLUSIONS

Allowing students to progress in their practice on simulators of increasing fidelity led to superior transfer of a broad range of clinical skills. Further, this progressive group was resource-efficient, as participants concentrated on lower fidelity and lower resource-intensive simulators. It is suggested that clinical training curricula incorporate exposure to multiple simulators to maximize educational benefit and potentially save educator time.

Perspective: authentic patient perspectives in simulations for procedural and surgical skills

In this article, the authors consider the role of the patient in simulation-based training and assessment of clinical procedural skills. In recent years, there has been a progressive shift of emphasis from teacher-centered to student-centered education, resulting in a redefinition of approaches to medical education. Traditional models of transmission of information from an expert to a novice have been supplanted by a more student-centered approach. However, medical education is not a matter for teacher and student alone. At the center is always the patient, around whom everything must ultimately rotate. A further shift is occurring. The patient is becoming the focal point of medical teaching and learning. It is argued that this shift is necessary and that simulation in its widest sense can be used to support this process. However, sensitivity to what we are simulating is essential, especially when simulations purport to address patient perspectives. The essay first reviews the history of medical education "centeredness," then outlines ways in which real and simulated patients are currently involved in medical education. Patient-focused simulation (PFS) is described as a means of offering patients' perspectives during the acquisition of clinical procedural and surgical skills. The authors draw on their experiences of developing PFS and preliminary work to "authenticate" simulations from patient perspectives. The essay concludes with speculation on the value of a "complementarity" model that acknowledges the authentic and equal perspectives of patients, students, clinicians, and teachers.