1
|
Nikolian VC, Camacho D, Earle D, Lehmann R, Nau P, Ramshaw B, Stulberg J. Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety. Surg Endosc 2024; 38:1583-1591. [PMID: 38332173 DOI: 10.1007/s00464-024-10677-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 12/30/2023] [Indexed: 02/10/2024]
Abstract
BACKGROUND Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric. METHODS The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2. RESULTS The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases. CONCLUSIONS A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.
Collapse
Affiliation(s)
- Vahagn C Nikolian
- Department of Surgery, Oregon Health & Science University, 3181 S.W. Sam Jackson Park Rd., Portland, OR, 97239, USA.
| | - Diego Camacho
- Minimally Invasive and Endoscopic Surgery at Montefiore Medical Center, New York, NY, USA
| | - David Earle
- New England Hernia Center, Lowell, MA, USA
- Tufts University School of Medicine, Boston, MA, USA
| | - Ryan Lehmann
- Department of Surgery, Section of Bariatric Surgery, University of Iowa Hospitals & Clinics, Iowa City, IA, USA
| | - Peter Nau
- Department of Surgery, Section of Bariatric Surgery, University of Iowa Hospitals & Clinics, Iowa City, IA, USA
| | - Bruce Ramshaw
- CQInsights PBC, Knoxville, TN, USA
- Caresyntax Corporation, Boston, MA, USA
| | - Jonah Stulberg
- Department of Surgery, McGovern Medical School University of Texas Health Science Center at Houston, Houston, TX, USA
| |
Collapse
|
2
|
Casey JC, Daniels AH. CORR Synthesis: How Have Film Review and Motion Analysis Been Used to Enhance Orthopaedic Surgical Performance? Clin Orthop Relat Res 2023; 481:564-579. [PMID: 36719752 PMCID: PMC9928675 DOI: 10.1097/corr.0000000000002506] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Accepted: 11/02/2022] [Indexed: 02/01/2023]
Affiliation(s)
- Jack C. Casey
- Division of Orthopaedics, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Alan H. Daniels
- Division of Orthopaedics, Warren Alpert Medical School of Brown University, Providence, RI, USA
| |
Collapse
|
3
|
Aldinc H, Gun C, Yaylaci S, Senuren CO, Guven F, Sahiner M, Kayayurt K, Turkmen S. Comparison of self versus expert-assisted feedback for cricothyroidotomy training: a randomized trial. BMC MEDICAL EDUCATION 2022; 22:455. [PMID: 35701782 PMCID: PMC9199165 DOI: 10.1186/s12909-022-03519-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/12/2021] [Accepted: 05/30/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND The self-video feedback method may have the potential to provide a low-cost alternative to physician-driven simulation-based training. This study aimed to assess the utility of two video feedback methods by comparing the improvement in performing cricothyroidotomy procedure following self video feedback (trainees review their performance by themselves) and expert-assisted video feedback (trainees review their performance while an emergency physician provides additional feedback). METHODS This study was pretest-posttest and two-group designed research performed at a university simulation center with 89 final-year medical students and used a cricothyroidotomy simulation model. After seeing an educational presentation and a best practice video, trainees were randomized into two groups; self video feedback group (SVFG) and expert-assisted video feedback group (EVFG). They performed the cricothyroidotomy before and after the feedback. The procedures were also recorded and scored by two emergency physicians. RESULTS There was a statistically significant improvement between pre-feedback and post-feedback assessments in terms of scores received and time needed for the procedures in both SVFG and EVFG groups (p < 0.05). Additionally, the post-feedback assessment scores were higher and time needed for the procedure was lower in the EVFG when compared with SVFG (p < 0.05 for both). CONCLUSIONS Results demonstrated significant improvement in cricothyroidotomy performance with both types of video feedback method. Even though the improvement was better in the EVFG compared to the SVFG, the self video feedback may have value especially in situations where expert-assisted feedback is not possible.
Collapse
Affiliation(s)
- Hasan Aldinc
- Department of Emergency Medicine, School of Medicine, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey
| | - Cem Gun
- Department of Emergency Medicine, School of Medicine, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey
| | - Serpil Yaylaci
- Department of Emergency Medicine, School of Medicine, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey
| | - Cigdem Ozkaya Senuren
- Department of First and Emergency Aid, Vocational School of Health Services, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey
| | - Feray Guven
- Center of Advanced Simulation and Education (CASE), Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey
| | - Melike Sahiner
- Department of Medical Education, School of Medicine, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey
| | - Kamil Kayayurt
- Department of Emergency Medicine, School of Medicine, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey
| | - Suha Turkmen
- Emergency Department, Hamad Medical Corporation, Doha, Qatar
| |
Collapse
|
4
|
Esposito AC, Coppersmith NA, White EM, Yoo PS. Video Coaching in Surgical Education: Utility, Opportunities, and Barriers to Implementation. JOURNAL OF SURGICAL EDUCATION 2022; 79:717-724. [PMID: 34972670 DOI: 10.1016/j.jsurg.2021.12.004] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Revised: 11/30/2021] [Accepted: 12/04/2021] [Indexed: 06/14/2023]
Abstract
OBJECTIVE This review discusses the literature on Video-Based Coaching (VBC) and explores the barriers to widespread implementation. DESIGN A search was performed on Scopus and PubMed for the terms "operation," "operating room," "surgery," "resident," "house staff," "graduate medical education," "teaching," "coaching," "assessment," "reflection," "camera," and "video" on July 27, 2021, in English. This yielded 828 results. A single author reviewed the titles and abstracts and eliminated any results that did not pertain to operative VBC or assessment. All bibliographies were reviewed, and appropriate manuscripts were included in this study. This resulted in a total of 52 manuscripts included in this review. SETTING/PARTICIPANTS Original, peer-reviewed studies focused on VBC or assessment. RESULTS VBC has been both subjectively and objectively found to be a valuable educational tool. Nearly every study of video recording in the operating room found that subjects, including surgical residents and seasoned surgeons alike, overwhelmingly considered it a useful, non-redundant adjunct to their training. Most studies that evaluated skill acquisition via standardized assessment tools found that surgical residents who underwent a VBC program had significant improvements compared to their counterparts who did not undergo video review. Despite this evidence of effectiveness, fewer than 5% of residency programs employ video recording in the operating room. Barriers to implementation include significant time commitments for proposed coaching curricula and difficulty with integration of video cameras into the operating room. CONCLUSIONS VBC has significant educational benefits, but a scalable curriculum has not been developed. An optimal solution would ensure technical ease and expediency, simple, high-quality cameras, immediate review, and overcoming entrenched surgical norms and culture.
Collapse
Affiliation(s)
- Andrew C Esposito
- Yale School of Medicine, Department of Surgery, New Haven, Connecticut
| | | | - Erin M White
- Yale School of Medicine, Department of Surgery, New Haven, Connecticut
| | - Peter S Yoo
- Yale School of Medicine, Department of Surgery, New Haven, Connecticut.
| |
Collapse
|
5
|
Kanters AE, Evilsizer SK, Regenbogen SE, Hendren S, Campbell DA, Dimick JB, Byrn JC. Correlation of Colorectal Surgical Skill With Patient Outcomes: A Cautionary Tale. Dis Colon Rectum 2022; 65:444-451. [PMID: 34840292 DOI: 10.1097/dcr.0000000000002124] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
BACKGROUND Previous work has demonstrated a correlation between video ratings of surgical skill and clinical outcomes. Some have proposed the use of video review for technical skill assessment, credentialing, and quality improvement. OBJECTIVE Before its adoption as a quality measure for colorectal surgeons, we must first determine whether video-based skill assessments can predict patient outcomes among specialty surgeons. DESIGN Twenty-one surgeons submitted one representative video of a minimally invasive colectomy. Each video was edited to highlight key steps and then rated by 10 peer surgeons using a validated American Society of Colon and Rectal Surgeons assessment tool. Linking surgeons' ratings to a validated surgical outcomes registry, we assessed the relationship between skill and risk-adjusted complication rates. SETTINGS The study was conducted with the Michigan Surgical Quality Collaborative, a statewide collaborative including 70 community, academic, and tertiary hospitals. PATIENTS Patients included those who underwent minimally invasive colorectal resection performed by the participating surgeons. MAIN OUTCOME MEASURES Main outcome measures included 30-day risk-adjusted postoperative complications. RESULTS The average technical skill rating for each surgeon ranged from 2.6 to 4.6. Risk-adjusted complication rate per surgeon ranged from 9.9% to 33.1%. Patients of surgeons in the bottom quartile of overall skill ratings were older and more likely to have hypertension or to smoke; patients of surgeons in the top quartile were more likely to be immunosuppressed or have an ASA score of 3 or higher. After patient- and surgery-specific risk adjustment, there was no statistically significant difference in complication rates between the bottom and top quartile surgeons (17.5% vs 16.8%, respectively, p = 0.41). LIMITATIONS Limitations included retrospective cohort design with short-term follow-up of sampled cases. Videos were edited to highlight key steps, and reviewers did not undergo training to establish norms. CONCLUSIONS Our study demonstrates that video-based peer rating of minimally invasive colectomy was not correlated with postoperative complications among specialty surgeons. As such, the adoption of video review for use in credentialing should be approached with caution. See Video Abstract at http://links.lww.com/DCR/B802.CORRELACIÓN ENTRE LA HABILIDAD QUIRÚRGICA COLORRECTAL Y LOS RESULTADOS OBTENIDOS EN EL PACIENTE: RELATO PRECAUTORIOANTECEDENTES:Trabajos anteriores han demostrado una correlación entre la video-calificación de la habilidad quirúrgica y los resultados clínicos. Algunos autores han propuesto el uso de la revisión de videos para la evaluación de la habilidad técnica, la acreditación y la mejoría en la calidad quirúrgica.OBJETIVO:Antes de su adopción como medida de calidad entre los cirujanos colorrectales, primero debemos determinar si las evaluaciones de habilidades basadas en video pueden predecir los resultados clínicos de los pacientes entre cirujanos especializados.DISEÑO:Veintiún cirujanos enviaron un video representativo de una colectomía mínimamente invasiva. Cada video fue editado para resaltar los pasos clave y luego fué calificado por 10 cirujanos revisores utilizando una herramienta de evaluación validada por la ASCRS. Al vincular las calificaciones de los cirujanos al registro de resultados quirúrgicos aprobado, evaluamos la relación entre la habilidad y las tasas de complicaciones ajustadas al riesgo.AJUSTE:Colaboración en todo el estado incluyendo 70 hospitales comunitarios, académicos y terciarios, el Michigan Surgical Quality Collaborative.PACIENTES:Todos aquellos sometidos a resección colorrectal mínimamente invasiva realizada por los cirujanos participantes.MEDIDA DE RESULTADO PRINCIPAL:Complicaciones posoperatorias ajustadas al riesgo a los 30 días.RESULTADOS:La calificación de la habilidad técnica promedio de cada cirujano osciló entre 2.6 y 4.6. La tasa de complicaciones ajustada al riesgo por cirujano osciló entre el 9,9% y el 33,1%. Los pacientes operados por los cirujanos del cuartil inferior de las calificaciones generales de habilidades eran fumadores y añosos, y tambiés más propensos a la hipertensión arterial. Los pacientes operados por los cirujanos del cuartil superior tenían más probabilidades de ser inmunosuprimidos o tener una puntuación ASA> = 3. Después del ajuste de riesgo específico de la cirugía y el paciente, no hubo diferencias estadísticamente significativas en las tasas de complicaciones entre los cirujanos del cuartil inferior y superior (17,5% frente a 16,8%, respectivamente, p = 0,41).LIMITACIONES:Diseño de cohortes retrospectivo con seguimiento a corto plazo de los casos muestreados. Los videos se editaron para resaltar los pasos clave y los revisores no recibieron capacitación para establecer normas.CONCLUSIONES:Nuestro estudio demuestra que la evaluación realizada por los revisores basada en el video de la colectomía mínimamente invasiva no se correlacionó con las complicaciones post-operatorias entre los cirujanos especialistas. Por tanto, la adopción de la revisión del video quirúrgico para su uso en la acreditación profesional, debe abordarse con mucha precaución. Consulte Video Resumen en http://links.lww.com/DCR/B802. (Traducción-Dr. Xavier Delgadillo).
Collapse
Affiliation(s)
- Arielle E Kanters
- Department of Surgery, Michigan Medicine, Ann Arbor, Michigan
- Center for Health Outcomes and Policy, University of Michigan, Ann Arbor, Michigan
| | | | - Scott E Regenbogen
- Department of Surgery, Michigan Medicine, Ann Arbor, Michigan
- Center for Health Outcomes and Policy, University of Michigan, Ann Arbor, Michigan
| | - Samantha Hendren
- Department of Surgery, Michigan Medicine, Ann Arbor, Michigan
- Center for Health Outcomes and Policy, University of Michigan, Ann Arbor, Michigan
| | | | - Justin B Dimick
- Department of Surgery, Michigan Medicine, Ann Arbor, Michigan
- Center for Health Outcomes and Policy, University of Michigan, Ann Arbor, Michigan
| | - John C Byrn
- Department of Surgery, Michigan Medicine, Ann Arbor, Michigan
- Center for Health Outcomes and Policy, University of Michigan, Ann Arbor, Michigan
| |
Collapse
|
6
|
James HK, Chapman AW, Pattison GTR, Fisher JD, Griffin DR. Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review. JBJS Rev 2021; 8:e1900167. [PMID: 33006464 PMCID: PMC7360100 DOI: 10.2106/jbjs.rvw.19.00167] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Abstract
BACKGROUND Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training. METHODS We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria. RESULTS One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent. CONCLUSIONS Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice. CLINICAL RELEVANCE Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.
Collapse
Affiliation(s)
- Hannah K James
- 1Clinical Trials Unit, Warwick Medical School, Coventry, United Kingdom 2Department of Trauma & Orthopedic Surgery, University Hospitals Coventry & Warwickshire, Coventry, United Kingdom
| | | | | | | | | |
Collapse
|
7
|
Sayari AJ, Chen O, Harada GK, Lopez GD. Success of Surgical Simulation in Orthopedic Training and Applications in Spine Surgery. Clin Spine Surg 2021; 34:82-86. [PMID: 33044270 DOI: 10.1097/bsd.0000000000001070] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/11/2019] [Accepted: 08/19/2020] [Indexed: 11/26/2022]
Abstract
STUDY DESIGN This was a narrative review. OBJECTIVE This study aimed to review the current literature on surgical simulation in orthopedics and its application to spine surgery. SUMMARY OF BACKGROUND DATA As orthopedic surgery increases in complexity, training becomes more relevant. There have been mandates in the United States for training orthopedic residents the fundamentals of surgical skills; however, few studies have examined the various training options available. Lack of funding, availability, and time are major constraints to surgical simulation options. METHODS A PubMed review of the current literature was performed on all relevant articles that examined orthopedic trainees using surgical simulation options. Studies were examined for their thoroughness and application of simulation options to orthopedic surgery. RESULTS Twenty-three studies have explored orthopedic surgical simulation in a setting that objectively assessed trainee performance, most in the field of trauma and arthroscopy. However, there was a lack of consistency in measurements made and skills tested by these simulators. There has only been one study exploring surgical simulation in spine surgery. CONCLUSIONS While there has been a growing number of surgical simulators to train orthopedic residents the fundamentals of surgical skills, most of these simulators are not feasible, reproducible, or available to the majority of training centers. Furthermore, the lack of consistency in the objective measurements of these studies makes interpretation of their results difficult. There is a need for more simulation in spine surgery, and future simulators and their respective studies should be reproducible, affordable, applicable to the surgical setting, and easily assembled by various programs across the world.
Collapse
Affiliation(s)
- Arash J Sayari
- Department of Orthopaedic Surgery, Rush University Medical Center, Chicago, IL
| | | | | | | |
Collapse
|
8
|
Halim J, Jelley J, Zhang N, Ornstein M, Patel B. The effect of verbal feedback, video feedback, and self-assessment on laparoscopic intracorporeal suturing skills in novices: a randomized trial. Surg Endosc 2020; 35:3787-3795. [PMID: 32804266 DOI: 10.1007/s00464-020-07871-3] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2020] [Accepted: 08/05/2020] [Indexed: 12/22/2022]
Abstract
BACKGROUND Laparoscopic skill acquisition involves a steep learning curve and laparoscopic suturing is an exceptionally challenging task. By improving the way feedback is given, trainees can learn these skills more effectively. This study aims to establish the most effective form of structured feedback on laparoscopic suturing skill acquisition in novices, by comparing the effects of expert verbal feedback, video review with expert feedback (video feedback), and video review with self-assessment. METHODS A prospective randomized blinded trial comparing verbal feedback, video feedback, and self-assessment. Novices in laparoscopic surgery were tasked with performing laparoscopic suturing with intracorporeal knot tying. Time was given for practice, and pre- and post-feedback assessments were undertaken. Suturing performance was measured using a task-specific checklist and global ratings. A post-study questionnaire was used to measure participant-perceived confidence, knowledge, and experience levels. RESULTS Fifty-one participants were randomized and allocated equally into the three groups. Performance in all three groups improved significantly from baseline. Video feedback had the largest improvement margin with checklist and global score improvements of 17.1% (± 9.9%) and 14.7% (± 9.3%), respectively. Performance improvements between groups were statistically significant in the global components (p = 0.004) but not the checklist components (p = 0.186). Global score improvement was significantly better in the video feedback group but was statistically insignificant between the self-assessment and verbal feedback groups. Questionnaire responses demonstrated positive results in confidence, knowledge, and experience levels, across all three study groups, with no differences between the groups (p > 0.05). CONCLUSION Structured video feedback facilitates reflection and self-directed learning, which improves the ability to develop proficiency in surgical skills. Combining both self-assessment and video feedback may be beneficial over verbal feedback alone due to the advantages of video review. These techniques should therefore be considered for implementation into surgical education curricula.
Collapse
Affiliation(s)
- Jonathan Halim
- Barts Cancer Institute, Queen Mary University of London, Charterhouse Square, London, UK.
| | - Joshua Jelley
- Barts Cancer Institute, Queen Mary University of London, Charterhouse Square, London, UK
| | - Ningning Zhang
- Barts Cancer Institute, Queen Mary University of London, Charterhouse Square, London, UK
| | - Marcus Ornstein
- Barts Cancer Institute, Queen Mary University of London, Charterhouse Square, London, UK
| | - Bijendra Patel
- Barts Cancer Institute, Queen Mary University of London, Charterhouse Square, London, UK
| |
Collapse
|
9
|
Green W, Shahzad MW, Wood S, Martinez Martinez M, Baines A, Navid A, Jay R, Whysall Z, Sandars J, Patel R. Improving junior doctor medicine prescribing and patient safety: An intervention using personalised, structured, video-enhanced feedback and deliberate practice. Br J Clin Pharmacol 2020; 86:2234-2246. [PMID: 32343422 PMCID: PMC7576627 DOI: 10.1111/bcp.14325] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2020] [Revised: 03/11/2020] [Accepted: 03/26/2020] [Indexed: 11/28/2022] Open
Abstract
AIMS This research investigated the effectiveness of an intervention for improving the prescribing and patient safety behaviour among Foundation Year doctors. The intervention consisted of simulated clinical encounters with subsequent personalised, structured, video-enhanced feedback and deliberate practice, undertaken at the start of four-month sub-specialty rotations. METHODS Three prospective, non-randomised control intervention studies were conducted, within two secondary care NHS Trusts in England. The primary outcome measure, error rate per prescriber, was calculated using daily prescribing data. Prescribers were grouped to enable a comparison between experimental and control conditions using regression analysis. A break-even analysis evaluated cost-effectiveness. RESULTS There was no significant difference in error rates of novice prescribers who received the intervention when compared with those of experienced prescribers. Novice prescribers not participating in the intervention had significantly higher error rates (P = .026, 95% confidence interval [CI] Wald 0.093 to 1.436; P = .026, 95% CI 0.031 to 0.397) and patients seen by them experienced significantly higher prescribing error rates (P = .007, 95% CI 0.025 to 0.157). Conversely, patients seen by the novice prescribers who received the intervention experienced a significantly lower rate of significant errors compared to patients seen by the experienced prescribers (P = .04, 95% CI -0.068 to -0.001). The break-even analysis demonstrates cost-effectiveness for the intervention. CONCLUSION Simulated clinical encounters using personalised, structured, video-enhanced feedback and deliberate practice improves the prescribing and patient safety behaviour of Foundation Year doctors. The intervention is cost-effective with potential to reduce avoidable harm.
Collapse
Affiliation(s)
- William Green
- University of Leicester School of Business, University of Leicester, Leicester, UK
| | | | - Stephen Wood
- University of Leicester School of Business, University of Leicester, Leicester, UK
| | - Maria Martinez Martinez
- Leicester General Hospital, University Hospitals of Leicester (UHL) NHS Trust, Leicester, UK
| | - Andrew Baines
- Pilgrim Hospital Boston, United Lincolnshire Hospitals (ULH) NHS Trust, Boston, Lincolnshire, UK
| | - Ahmad Navid
- Queen Elizabeth Hospital Birmingham, University Hospitals Birmingham NHS Foundation Trust, Edgbaston, Birmingham, UK
| | - Robert Jay
- School of Medicine, University of Nottingham, Nottingham, UK
| | - Zara Whysall
- Department of Human Resource Management, Nottingham Business School, Nottingham Trent University, Nottingham, UK
| | - John Sandars
- Health Research Institute, Faculty of Health, Social Care and Medicine, Edge Hill University, Ormskirk, Lancashire, UK
| | - Rakesh Patel
- School of Medicine, University of Nottingham, Nottingham, UK
| |
Collapse
|
10
|
Yu J, Lo C, Madampage C, Bajwa J, O'Brien J, Olszynski P, Lucy M. Video Modeling and Video Feedback to Reduce Time to Perform Intravenous Cannulation in Medical Students: A Randomized-Controlled Mixed-Methods Study. Can J Anaesth 2020; 67:715-725. [PMID: 32052372 DOI: 10.1007/s12630-020-01570-2] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2019] [Revised: 12/16/2019] [Accepted: 12/18/2019] [Indexed: 11/28/2022] Open
Abstract
PURPOSE Combined video modeling (VM) and video feedback (VF) may be more beneficial than traditional feedback when teaching procedural skills. This study examined whether repeated VM and VF compared with VM alone reduced the time required for medical students to perform peripheral intravenous (IV) cannulation. METHODS Twenty-five novice medical students were randomly assigned to groups in a one-way blinded embedded mixed-methods study to perform IV cannulation. Participants received standardized instruction and performed IV cannulation on each other while being audio-video recorded. They were assigned to review a video of an expert performing IV cannulation (VM alone), or both the expert video and a video of their own most recent IV cannulation (VM+VF), before returning to perform another IV cannulation. This was repeated for a total of four IV cannulation encounters and three video reviews. A post-test interview was also conducted and analyzed qualitatively using thematic content analysis. RESULTS The median [interquartile range] time required to perform IV cannulation in the final encounter was significantly different between the VM+VF group vs VM alone group (126 [93-226] sec vs 345 [131-537] sec, respectively; median difference, 111 sec; 95% confidence interval, 8 to 391; P = 0.02). There was no significant difference in IV cannulation success between VM alone and VM+VF in the final encounter (75% vs 85% respectively; P = 0.65). For the VM+VF group, the time to perform IV cannulation was reduced after the final encounter compared with the baseline encounter (P = 0.002), which was not true of the VM alone group (P = 0.35). CONCLUSION Video modeling and feedback shortened time to IV skill completion, reduced complications, and improved satisfaction in novice medical students.
Collapse
Affiliation(s)
- Julie Yu
- Department of Anesthesiology, Perioperative Medicine and Pain Management, Royal University Hospital, University of Saskatchewan, G525-103 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada.
| | - Calvin Lo
- Department of Anesthesiology, Perioperative Medicine and Pain Management, Royal University Hospital, University of Saskatchewan, G525-103 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Claudia Madampage
- Department of Anesthesiology, Perioperative Medicine and Pain Management, Royal University Hospital, University of Saskatchewan, G525-103 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Jagmeet Bajwa
- Department of Anesthesiology, Perioperative Medicine and Pain Management, Royal University Hospital, University of Saskatchewan, G525-103 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Jennifer O'Brien
- Department of Anesthesiology, Perioperative Medicine and Pain Management, Royal University Hospital, University of Saskatchewan, G525-103 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Paul Olszynski
- Department of Emergency Medicine, Royal University Hospital, University of Saskatchewan, Saskatoon, SK, Canada
| | - Malcolm Lucy
- Department of Anesthesiology, Perioperative Medicine and Pain Management, Royal University Hospital, University of Saskatchewan, G525-103 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| |
Collapse
|
11
|
Abstract
Approximately 1 in 10 newborns will require basic resuscitation interventions at birth. Some infants progress to require more advanced measures including the provision of positive pressure ventilation, chest compressions, intubation and administration of volume/cardiac medications. Although advanced resuscitation is infrequent, it is crucial that personnel adequately trained in these techniques are available to provide such resuscitative measures. In 2000, Louis Halmalek et al. called for a "New Paradigm in Pediatric Medical Education: Teaching Neonatal Resuscitation in a Simulated Delivery Room Environment." This was one of the first articles to highlight simulation as a method of teaching newborn resuscitation. The last decades have seen an exponential growth in the area of simulation in newborn care, in particular in newborn resuscitation and stabilization. Simulation is best defined as an instructional strategy "used to replace or amplify real experiences with guided experiences that evoke or replicate substantial aspects of the real world in a fully interactive manner." Simulation training has now become an important point of how we structure training and deliver improved healthcare to patients. Some of the key aspects of simulation training include feedback, deliberate practice, outcome measurement, retention of skills and curriculum integration. The term "Train to win" is often used in sporting parlance to define how great teams succeed. The major difference between sports teams is that generally their game day comes once a week, whereas in newborn resuscitation every day is potentially "game day." In this review we aim to summarize the current evidence on the use of simulation based education and training in neonatal resuscitation, with particular emphasis on the evidence supporting its effectiveness. We will also highlight recent advances in the development of simulation based medical education in the context of newborn resuscitation to ensure we "train to win."
Collapse
Affiliation(s)
- Aisling A Garvey
- Department of Paediatrics and Child Health, Neonatal Intensive Care Unit, University College Cork, Cork, Ireland.,INFANT Research Centre, Cork, Ireland
| | - Eugene M Dempsey
- Department of Paediatrics and Child Health, Neonatal Intensive Care Unit, University College Cork, Cork, Ireland.,INFANT Research Centre, Cork, Ireland
| |
Collapse
|
12
|
McQueen S, McKinnon V, VanderBeek L, McCarthy C, Sonnadara R. Video-Based Assessment in Surgical Education: A Scoping Review. JOURNAL OF SURGICAL EDUCATION 2019; 76:1645-1654. [PMID: 31175065 DOI: 10.1016/j.jsurg.2019.05.013] [Citation(s) in RCA: 36] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/28/2019] [Revised: 05/15/2019] [Accepted: 05/18/2019] [Indexed: 06/09/2023]
Abstract
BACKGROUND AND OBJECTIVE Video-based assessment of residents' surgical skills may offer several advantages over direct observations of clinical performance in terms of objectivity, time-efficiency, and feasibility. Although video-based assessment is becoming more common in surgical training, a broad understanding of its utility is lacking. This scoping review explores video-based assessment in surgical training and presents the evidence supporting its use. DESIGN A literature search was conducted using the Web of Science database with key words related to video-based assessment and surgical training. Exclusion criteria included articles not published in English and articles on undergraduate medical education, continuing professional development, or non-surgical disciplines. Initially, 702 articles were identified; after title, abstract, and full-text screening by two independent reviewers (SM and VM), 199 articles remained. RESULTS We present the benefits of video-based assessment, including the ability to capture clinical ability in the operating room without decreasing intraoperative efficiency, as well as the potential to improve formative assessment and feedback practices. We describe the validity, reliability, and challenges of video-based assessment, as well as the use of video-based methods in clinical and simulated settings. We conclude by discussing questions that remain to be addressed. CONCLUSIONS Although further research and cost-benefit analyses are required, greater adoption of video-based assessment into surgical training may help meet increased assessment demands in an era of competency-based medical education.
Collapse
Affiliation(s)
- Sydney McQueen
- Department of Surgery, University of Toronto, Toronto, Ontario, Canada
| | - Victoria McKinnon
- Department of Surgery, McMaster University, Hamilton, Ontario, Canada
| | - Laura VanderBeek
- Department of Surgery, McMaster University, Hamilton, Ontario, Canada
| | - Colm McCarthy
- Department of Surgery, McMaster University, Hamilton, Ontario, Canada
| | - Ranil Sonnadara
- Department of Surgery, University of Toronto, Toronto, Ontario, Canada; Department of Surgery, McMaster University, Hamilton, Ontario, Canada.
| |
Collapse
|
13
|
Farooq A, Reso A, Harvey A. Instant replay: Evaluation of instant video feedback in surgical novices for a laparoscopic gallbladder dissection. Am J Surg 2019; 215:943-947. [PMID: 29395018 DOI: 10.1016/j.amjsurg.2018.01.013] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2017] [Revised: 01/06/2018] [Accepted: 01/12/2018] [Indexed: 11/28/2022]
Abstract
BACKGROUND Athletes often use video to improve their technique. We hypothesized that surgical novices given feedback using video-replay would outperform surgical novices given verbal feedback in the performance of a laparoscopic task. METHODS Our study used a prospective, randomized control design. The surgical task involved the laparoscopic dissection of a pig gallbladder. Our participants performed a dissection, pre- and post-traditional or video feedback. Each recording was independently scored by two staff surgeons using the previously validated rating tools. RESULTS There was no significant difference between video feedback or traditional feedback groups in their mean overall or task specific scores. Both traditional and video-feedback groups had a trend towards improved performance post-feedback. CONCLUSIONS No significant difference in performance by both our global assessment metrics or task-specific metrics was observed. Video feedback requires further study to investigate its impact on surgical training.
Collapse
Affiliation(s)
- Ameer Farooq
- Section of General Surgery, Department of Surgery, University of Calgary, Room 1023, North Tower, 1403 - 29 St, T2N 2T9, Calgary, AB, Canada
| | - Artan Reso
- Section of General Surgery, Department of Surgery, University of Calgary, Room 1023, North Tower, 1403 - 29 St, T2N 2T9, Calgary, AB, Canada.
| | - Adrian Harvey
- Section of General Surgery, Department of Surgery, Room 1005, North Tower, 1403 - 29 St, T2N 2T9, Calgary, AB, Canada.
| |
Collapse
|
14
|
Dickerson P, Grande S, Evans D, Levine B, Coe M. Utilizing Intraprocedural Interactive Video Capture With Google Glass for Immediate Postprocedural Resident Coaching. JOURNAL OF SURGICAL EDUCATION 2019; 76:607-619. [PMID: 30833204 DOI: 10.1016/j.jsurg.2018.10.002] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/20/2018] [Revised: 10/02/2018] [Accepted: 10/07/2018] [Indexed: 06/09/2023]
Abstract
BACKGROUND Video coaching has been found to be an effective teaching method because it incorporates many of the established principles of successful adult learning. The goal of our study was to assess the feasibility and effectiveness of using a point-of-view video camera (Google Glass) to improve the surgical skills education of orthopaedic surgery residents. METHODS Forty-two residents from 4 institutions participated in a partially blinded randomized control trial performing an intra-articular distal tibial fracture reduction task while wearing Google Glass to record the performance. Participants underwent a structured coaching session with 20 participants (intervention group) using the recorded video to augment this session, and 22 participants (control group) receiving verbal coaching alone. The task was repeated again immediately after the coaching session. Performance was scored using an Objective Structured Assessment of Technical Skills checklist, Global Rating Scale, fluoroscopic usage, and reduction quality. A semistructured interview was then performed to assess experience of participants. RESULTS There was no significant difference (p > 0.05) seen in score improvement in the Objective Structured Assessment of Technical Skills checklist, Global Rating Scale, fluoroscopic usage, or reduction quality between the control and intervention groups. Thematic analysis of interview showed majority of participants found video coaching increased effectiveness in understanding of goals, developing techniques and strategies, and process of self-reflection. Their involvement was seen overall as a positive experience, with participants wanting to see more inclusion of video coaching within surgical education. CONCLUSIONS No difference in performance improvement between the 2 groups was seen, but majority of participants found the video coaching sessions valuable and could have potential beneficial role in education.
Collapse
Affiliation(s)
| | - Stuart Grande
- Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire
| | | | | | - Marcus Coe
- Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire
| |
Collapse
|
15
|
Naik ND, Abbott EF, Gas BL, Murphy BL, Farley DR, Cook DA. Personalized video feedback improves suturing skills of incoming general surgery trainees. Surgery 2018; 163:921-926. [DOI: 10.1016/j.surg.2017.11.006] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2017] [Accepted: 11/07/2017] [Indexed: 10/18/2022]
|
16
|
Abbott EF, Thompson W, Pandian TK, Zendejas B, Farley DR, Cook DA. Personalized Video Feedback and Repeated Task Practice Improve Laparoscopic Knot-Tying Skills: Two Controlled Trials. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2017; 92:S26-S32. [PMID: 29065020 DOI: 10.1097/acm.0000000000001924] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
PURPOSE Compare the effect of personalized feedback (PF) vs. task demonstration (TD), both delivered via video, on laparoscopic knot-tying skills and perceived workload; and evaluate the effect of repeated practice. METHOD General surgery interns and research fellows completed four repetitions of a simulated laparoscopic knot-tying task at one-month intervals. Midway between repetitions, participants received via e-mail either a TD video (demonstration by an expert) or a PF video (video of their own performance with voiceover from a blinded senior surgeon). Each participant received at least one video per format, with sequence randomly assigned. Outcomes included performance scores and NASA Task Load Index (NASA-TLX) scores. To evaluate the effectiveness of repeated practice, scores from these trainees on a separate delayed retention test were compared against historical controls who did not have scheduled repetitions. RESULTS Twenty-one trainees completed the randomized study. Mean change in performance scores was significantly greater for those receiving PF (difference = 23.1 of 150 [95% confidence interval (CI): 0, 46.2], P = .05). Perceived workload was also significantly reduced (difference = -3.0 of 20 [95% CI: -5.8, -0.3], P = .04). Compared with historical controls (N = 93), the 21 with scheduled repeated practice had higher scores on the laparoscopic knot-tying assessment two weeks after the final repetition (difference = 1.5 of 10 [95% CI: 0.2, 2.8], P = .02). CONCLUSIONS Personalized video feedback improves trainees' procedural performance and perceived workload compared with a task demonstration video. Brief monthly practice sessions support skill acquisition and retention.
Collapse
Affiliation(s)
- Eduardo F Abbott
- E.F. Abbott is a simulation fellow, Mayo Clinic Multidisciplinary Simulation Center, Mayo Clinic College of Medicine, Rochester, Minnesota, and adjunct instructor of internal medicine, Department of Internal Medicine, Escuela de Medicina, Pontificia Universidad Catolica de Chile, Santiago, Chile; ORCID: http://orcid.org/0000-0001-5713-4809. W. Thompson is a medical student, University of Minnesota Medical School, Minneapolis, Minnesota. T.K. Pandian is a resident, Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota. B. Zendejas is a pediatric surgery fellow, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts. D.R. Farley is professor of surgery and consultant, Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota. D.A. Cook is professor of medicine and professor of medical education; research chair, Mayo Clinic Multidisciplinary Simulation Center; director of research, Office of Applied Scholarship and Education Science; and Consultant in the Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota; ORCID: http://orcid.org/0000-0003-2383-4633
| | | | | | | | | | | |
Collapse
|
17
|
Hamstra SJ, Dubrowski A. Effective Training and Assessment of Surgical Skills, and the Correlates of Performance. Surg Innov 2016; 12:71-7. [PMID: 15846449 DOI: 10.1177/155335060501200110] [Citation(s) in RCA: 33] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
Abstract
This report briefly describes the University of Toronto Surgical Skills Centre and summarizes research in technical skills conducted at that site. This includes work on curriculum evaluation, the development and validation of assessment instruments, the retention of technical skills after training, and the prediction of success in surgery. These studies benefited from the large number of participants made available by the Surgical Skills Centre, allowing for randomized controlled studies or correlation studies where larger numbers are necessary for adequate statistical power. Recent emphasis has been on the further development of objective means of assessment and the exploration of correlates of surgical performance. Ongoing research is also aimed at simulator validation.
Collapse
Affiliation(s)
- Stanley J Hamstra
- Department of Surgery and the Wilson Centre for Research in Education, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
| | | |
Collapse
|
18
|
Vassiliou MC, Feldman LS, Fraser SA, Charlebois P, Chaudhury P, Stanbridge DD, Fried GM. Evaluating Intraoperative Laparoscopic Skill: Direct Observation Versus Blinded Videotaped Performances. Surg Innov 2016; 14:211-6. [DOI: 10.1177/1553350607308466] [Citation(s) in RCA: 48] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
The Global Operative Assessment of Laparoscopic Skill (GOALS) has been shown to meet high standards for direct observation. The purpose of this study was to investigate the reliability and validity of GOALS when applied to blinded, videotaped performances. Five novice surgeons and 5 experienced surgeons were each evaluated by 2 observers during a laparoscopic cholecystectomy. Subsequently, 4 laparoscopists (V1 to V4) evaluated the videotaped procedures using GOALS. Two of the raters (V1 and V3) had prior experience using GOALS. The interrater reliabilities between video raters (VRs) and between VRs and direct raters (DRs) were calculated using the intraclass correlation coefficient. Construct validity was assessed using 2-way analysis of variance. Interrater reliability between the 4 VRs and the 2 DRs was 0.72. The intraclass correlation coefficient for the 4 VRs was 0.68 and for each VR compared with the mean DR was 0.86, 0.39, 0.94, and 0.76, respectively. All raters, except V2, differentiated between novice and experienced groups ( P values ranged from .01 to .05). These data suggest that GOALS can be used to assess laparoscopic skill based on videotaped performances but that rater training may play an important role in ensuring the reliability and validity of the instrument. Experience with the tool in the operating room may improve the reliability of video rating and could be of value in training evaluators.
Collapse
Affiliation(s)
- Melina C. Vassiliou
- Department of Surgery, McGill University, Montreal, Canada, Steinberg-Bernstein Centre for Minimally Invasive Surgery McGill University, Montreal, Canada
| | - Liane S. Feldman
- Department of Surgery, McGill University, Montreal, Canada, Steinberg-Bernstein Centre for Minimally Invasive Surgery McGill University, Montreal, Canada
| | - Shannon A. Fraser
- Department of Surgery, McGill University, Montreal, Canada, Steinberg-Bernstein Centre for Minimally Invasive Surgery McGill University, Montreal, Canada
| | | | | | - Donna D. Stanbridge
- Steinberg-Bernstein Centre for Minimally Invasive Surgery McGill University, Montreal, Canada
| | - Gerald M. Fried
- Department of Surgery, McGill University, Montreal, Canada, , Steinberg-Bernstein Centre for Minimally Invasive Surgery McGill University, Montreal, Canada
| |
Collapse
|
19
|
Wittler M, Hartman N, Manthey D, Hiestand B, Askew K. Video-augmented feedback for procedural performance. MEDICAL TEACHER 2016; 38:607-612. [PMID: 26383586 DOI: 10.3109/0142159x.2015.1075650] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
BACKGROUND Resident programs must assess residents' achievement of core competencies for clinical and procedural skills. AIMS Video-augmented feedback may facilitate procedural skill acquisition and promote more accurate self-assessment. METHODS A randomized controlled study to investigate whether video-augmented verbal feedback leads to increased procedural skill and improved accuracy of self-assessment compared to verbal only feedback. Participants were evaluated during procedural training for ultrasound guided internal jugular central venous catheter (US IJ CVC) placement. All participants received feedback based on a validated 30-point checklist for US IJ CVC placement and validated 6-point procedural global rating scale. RESULTS Scores in both groups improved by a mean of 9.6 points (95% CI: 7.8-11.4) on the 30-point checklist, with no difference between groups in mean score improvement on the global rating scale. In regards to self-assessment, participant self-rating diverged from faculty scoring, increasingly so after receiving feedback. Residents rated highly by faculty underestimated their skill, while those rated more poorly demonstrated increasing overestimation. CONCLUSIONS Accuracy of self-assessment was not improved by addition of video. While feedback advanced the skill of the resident, video-augmented feedback did not enhance skill acquisition or improve accuracy of resident self-assessment compared to standard feedback.
Collapse
Affiliation(s)
- Mary Wittler
- a Department of Emergency Medicine , Wake Forest Baptist Medical Center , USA
| | - Nicholas Hartman
- a Department of Emergency Medicine , Wake Forest Baptist Medical Center , USA
| | - David Manthey
- a Department of Emergency Medicine , Wake Forest Baptist Medical Center , USA
| | - Brian Hiestand
- a Department of Emergency Medicine , Wake Forest Baptist Medical Center , USA
| | - Kim Askew
- a Department of Emergency Medicine , Wake Forest Baptist Medical Center , USA
| |
Collapse
|
20
|
Junod Perron N, Louis-Simonet M, Cerutti B, Pfarrwaller E, Sommer J, Nendaz M. Feedback in formative OSCEs: comparison between direct observation and video-based formats. MEDICAL EDUCATION ONLINE 2016; 21:32160. [PMID: 27834170 PMCID: PMC5103667 DOI: 10.3402/meo.v21.32160] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/06/2016] [Revised: 09/19/2016] [Accepted: 09/21/2016] [Indexed: 05/12/2023]
Abstract
INTRODUCTION Medical students at the Faculty of Medicine, University of Geneva, Switzerland, have the opportunity to practice clinical skills with simulated patients during formative sessions in preparation for clerkships. These sessions are given in two formats: 1) direct observation of an encounter followed by verbal feedback (direct feedback) and 2) subsequent review of the videotaped encounter by both student and supervisor (video-based feedback). The aim of the study was to evaluate whether content and process of feedback differed between both formats. METHODS In 2013, all second- and third-year medical students and clinical supervisors involved in formative sessions were asked to take part in the study. A sample of audiotaped feedback sessions involving supervisors who gave feedback in both formats were analyzed (content and process of the feedback) using a 21-item feedback scale. RESULTS Forty-eight audiotaped feedback sessions involving 12 supervisors were analyzed (2 direct and 2 video-based sessions per supervisor). When adjusted for the length of feedback, there were significant differences in terms of content and process between both formats; the number of communication skills and clinical reasoning items addressed were higher in the video-based format (11.29 vs. 7.71, p=0.002 and 3.71 vs. 2.04, p=0.010, respectively). Supervisors engaged students more actively during the video-based sessions than during direct feedback sessions (self-assessment: 4.00 vs. 3.17, p=0.007; active problem-solving: 3.92 vs. 3.42, p=0.009). Students made similar observations and tended to consider that the video feedback was more useful for improving some clinical skills. CONCLUSION Video-based feedback facilitates discussion of clinical reasoning, communication, and professionalism issues while at the same time actively engaging students. Different time and conceptual frameworks may explain observed differences. The choice of feedback format should depend on the educational goal.
Collapse
Affiliation(s)
- Noëlle Junod Perron
- Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Geneva, Switzerland
- Division of Primary Care Medicine, Department of Community Medicine, Primary Care and Emergencies, Geneva University Hospitals, Geneva, Switzerland;
| | - Martine Louis-Simonet
- Service of General Internal Medicine, Department of General Internal Medicine, Rehabilitation and Geriatric Medicine, Geneva University Hospitals, Geneva, Switzerland
| | - Bernard Cerutti
- Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| | - Eva Pfarrwaller
- Unit of Primary Care Teaching and Research, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| | - Johanna Sommer
- Unit of Primary Care Teaching and Research, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| | - Mathieu Nendaz
- Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Geneva, Switzerland
- Service of General Internal Medicine, Department of General Internal Medicine, Rehabilitation and Geriatric Medicine, Geneva University Hospitals, Geneva, Switzerland
| |
Collapse
|
21
|
Nesbitt CI, Phillips AW, Searle RF, Stansby G. Randomized trial to assess the effect of supervised and unsupervised video feedback on teaching practical skills. JOURNAL OF SURGICAL EDUCATION 2015; 72:697-703. [PMID: 25703737 DOI: 10.1016/j.jsurg.2014.12.013] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/02/2014] [Revised: 12/15/2014] [Accepted: 12/29/2014] [Indexed: 06/04/2023]
Abstract
BACKGROUND Feedback is a vital component of the learning process; however, great variation exists in the quality, quantity, and method of delivery. Video feedback is not commonly used in the teaching of surgical skills. The aim of this trial was to evaluate the benefit of 2 types of video feedback-individualized video feedback (IVF), with the student reviewing their performance with an expert tutor, and unsupervised video-enhanced feedback (UVF), where the student reviews their own performance together with an expert teaching video-to determine if these improve performance when compared with a standard lecture feedback. METHODS A prospective blinded randomized control trial comparing lecture feedback with IVF and UVF was carried out. Students were scored by 2 experts directly observing the performance and 2 blinded experts using a validated pro forma. Participants were recorded on video when performing a suturing task. They then received their feedback via any of the 3 methods before being invited to repeat the task. RESULTS A total of 32 students were recruited between the 3 groups. There was no significant difference in suturing skill performance scores given by those directly observing the students and those blinded to the participant. There was no statistically significant difference between the 2 video feedback groups (p = 1.000), but there was significant improvement between standard lecture feedback and UVF (p = 0.047) and IVF (p = 0.001). CONCLUSION Video feedback can facilitate greater learning of clinical skills. Students can attain a similar level of surgical skills improvement with UVF as with teacher-intensive IVF.
Collapse
Affiliation(s)
- Craig I Nesbitt
- Northern Vascular Centre, Freeman Hospital, Newcastle upon Tyne, United Kingdom.
| | - Alexander W Phillips
- Northern Oesophagogastric Centre, Royal Victoria Infirmary, Newcastle upon Tyne, United Kingdom
| | - Roger F Searle
- School of Medical Sciences Education Development, Newcastle University, Newcastle upon Tyne, United Kingdom
| | - Gerard Stansby
- Northern Vascular Centre, Freeman Hospital, Newcastle upon Tyne, United Kingdom
| |
Collapse
|
22
|
Determining the Role of a National Objective Assessment of Surgical Skills in Gynecological Oncology: An e-Delphi Methodology. Int J Gynecol Cancer 2014; 24:1098-104. [DOI: 10.1097/igc.0000000000000157] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/01/2023] Open
Abstract
ObjectiveThe aim of this study is to determine a national consensus on the role of an objective assessment of technical surgical skills in gynecological oncology (GO).MethodsAfter approval was obtained from Society of Gynecologic Oncology of Canada, A panel of 20 GO leaders was assembled, representing all GO fellowship programs, and was asked to participate in an anonymous group and respond to an online 49-item questionnaire using a modified Delphi methodology.ResultsNinety-five percent (n = 19) of those invited to participate did so. Seventeen of the panelists (89.5%) believed there was no sufficiently standardized technical skills assessment for GO fellows, whereas 18 responders (95%) believed that fellows should be objectively assessed on more than 1 occasion during their training. Consensus was predefined as Cronbach α greater than 0.8. The panel agreed on what procedures should be objectively assessed with a Cronbach α of 0.967. An overall Cronbach α of 0.993 was achieved after a single Delphi round.ConclusionsWe achieved consensus on the possible components and logistics of a skills assessment process among a group of highly experienced GO trainers in Canada. This study provides the basis for further investigation and debate on the potential value, necessity, and feasibility of an assessment of advanced surgical and nonsurgical skills of GO trainees.
Collapse
|
23
|
Farquharson AL, Cresswell AC, Beard JD, Chan P. Randomized trial of the effect of video feedback on the acquisition of surgical skills. Br J Surg 2013; 100:1448-53. [DOI: 10.1002/bjs.9237] [Citation(s) in RCA: 45] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/20/2013] [Indexed: 11/12/2022]
Abstract
Abstract
Background
Constructive feedback provides a mechanism for reinforcing learning during the acquisition of surgical skills. Feedback is usually given verbally, and sometimes documented, after direct observation by a trained assessor. The aim was to evaluate video recording as an effective modality for enhancing feedback, in comparison with standard verbal feedback alone.
Methods
This was a prospective, blinded, randomized clinical trial comparing standard verbal feedback plus video with standard verbal feedback alone. Validated pro formas for assessment were used and quality control was performed by independent expert assessors. Trial participants were recorded on video performing the surgical skill, and returned the next day to perform the skill again following video and standard verbal feedback (group 1) or standard verbal feedback alone (group 2).
Results
Forty-eight participants were divided equally between the two groups. There was a significant improvement in the mean overall procedure score for group 1 of 2·875 from a maximum achievable score of 20 (P = 0·003), but not for group 2. There were significant improvements in the specific domains of instrument familiarity, needle handling, skin handling and accurate apposition, again all in group 1. The only significant improvement in group 2 was in an organized approach to the task, also observed in group 1. Knot-tying security deteriorated after feedback in group 2 but not in group 1.
Conclusion
The addition of video feedback can improve the acquisition of surgical skills, and could be incorporated into formal surgical curricula.
Collapse
Affiliation(s)
- A L Farquharson
- Colorectal Surgical Unit, University of Sheffield, Sheffield, UK
- Academic Unit of Medical Education, University of Sheffield, Sheffield, UK
| | - A C Cresswell
- Colorectal Surgical Unit, University of Sheffield, Sheffield, UK
| | - J D Beard
- Sheffield Vascular Institute, Northern General Hospital, Sheffield Teaching Hospitals NHS Foundation Trust, University of Sheffield, Sheffield, UK
- Academic Unit of Medical Education, University of Sheffield, Sheffield, UK
| | - P Chan
- Sheffield Vascular Institute, Northern General Hospital, Sheffield Teaching Hospitals NHS Foundation Trust, University of Sheffield, Sheffield, UK
- Academic Unit of Medical Education, University of Sheffield, Sheffield, UK
| |
Collapse
|
24
|
A comparison C1-C2 transarticular screw placement after self-education and mentored education of orthopaedic residents. ACTA ACUST UNITED AC 2013; 25:E155-60. [PMID: 22576717 DOI: 10.1097/bsd.0b013e31825bd0f6] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
STUDY DESIGN Prospective randomized trial. OBJECTIVE This study will provide preliminary data on whether residents can be "self-taught" and to what extent a lecture, demonstration, and coaching can improve skills and knowledge. BACKGROUND DATA Practice-based learning is an essential competency in Accreditation Council for graduate Medical Education-accredited residencies. Little has been done to demonstrate whether residents can be self-taught or benefit from mentoring in understanding and performing difficult surgical tasks. METHODS A written test was given to orthopedic residents on C1-C2 transarticular screw placement. They were then provided reading on C1-C2 transarticular screw placement. Residents were then divided into a "self-directed learning" group and a "mentored learning" group. All residents then performed the technique on models, with the "mentored" group receiving a lecture and coaching from the mentor. The models were analyzed for technique errors and the previous test was administered again as a posttest. The test and screw placement were repeated 4 months later. RESULTS Residents without mentoring had an average improvement of 4.5 points, those with mentoring had average improvement of 8.6 points (P=0.0068). The screw placement technique error rate for the nonmentored group (n=8) was 2.55, and for the mentored group (n=9) was 1.47 (P=0.004). Sixteen residents completed the delayed test, 7 from the nonmentored groups and 8 from the mentored group. Nine residents were able to repeat the screw placement technique 4 months after the initial test and screw placement, 3 nonentored, and 6 mentored. Although there were some trends toward the mentored group having better retention, neither knowledge nor skill was statistically different. CONCLUSIONS This preliminary trial seems to indicate that residents provided a lecture and guided technical instruction will obtain knowledge and perform procedures better than those that do not. Conclusions based upon Post Graduate year, motivation, and interest in spine surgery could not be made.
Collapse
|
25
|
Hamstra SJ. Keynote address: the focus on competencies and individual learner assessment as emerging themes in medical education research. Acad Emerg Med 2012; 19:1336-43. [PMID: 23279242 DOI: 10.1111/acem.12021] [Citation(s) in RCA: 18] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2012] [Accepted: 07/06/2012] [Indexed: 12/01/2022]
Abstract
This article describes opportunities for scholarship in medical education, based on a brief overview of recent changes in medical education. The implications arising from these changes are discussed, with recommendations for focus, and suggestions and examples for making progress in this field. The author discusses 1) the historical context of the current shift toward competency-based medical education, 2) the potential contribution of social and behavioral sciences to medical education scholarship, 3) methods and approaches for supporting scholarship in medical education, and very briefly 4) trends in simulation. The author concludes with a call for quality in medical education scholarship and argues that the most promising and fruitful area of medical education scholarship for the future lies in the field of assessment of individual competence.
Collapse
Affiliation(s)
- Stanley J. Hamstra
- Academy for Innovation in Medical Education (AIME); University of Ottawa Skills and Simulation Centre; Departments of Medicine, Surgery, and Anesthesia; Faculty of Medicine University of Ottawa; Ottawa; Ontario; Canada
| |
Collapse
|
26
|
VanHeest A, Kuzel B, Agel J, Putnam M, Kalliainen L, Fletcher J. Objective structured assessment of technical skill in upper extremity surgery. J Hand Surg Am 2012; 37:332-7. 337.e1-4. [PMID: 22281169 DOI: 10.1016/j.jhsa.2011.10.050] [Citation(s) in RCA: 36] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/02/2011] [Revised: 10/21/2011] [Accepted: 10/25/2011] [Indexed: 02/02/2023]
Abstract
PURPOSE Objective assessment of technical skills in hand surgery has been lacking. This article reports on an Objective Structured Assessment of Technical Skills format of a multiple bench-station evaluation of orthopedic surgery residents' technical skills for 3 common upper extremity surgeries. METHODS Twenty-seven residents (6 postgraduate year [PGY] 2, 8 PGY 3, 8 PGY 4, and 5 PGY 5) participated in the examination. Each resident performed surgery on a cadaveric specimen at 3 stations, trigger finger release (TFR), open carpal tunnel release, and distal radius fracture fixation. A board-certified hand surgeon evaluated trainee performance at each station, using a procedure-specific detailed checklist, a validated global rating scale, and pass/fail assessment. A resident post-testing evaluation was collected. RESULTS Construct validity with correlation between year in training and detailed checklist scores was demonstrated for TFR and carpal tunnel release; between year in training and global rating scores for TFR and distal radius fracture fixation; and between year in training and pass/fail assessment for TFR. Criterion validity was demonstrated by the correlation between global rating scale scores, detailed checklist scores, and pass/fail assessment for TFR, carpal tunnel release, and distal radius fracture fixation. Time to complete the surgery was not correlated with surgical performance. Residents rated the multiple-station Objective Structured Assessment of Technical Skills format as highly educational. CONCLUSIONS This study reports that a surgeon's ability to release a trigger finger does not correlate specifically to his or her ability to perform a carpal tunnel release or to perform plate fixation of a radius fracture. The results of this study would indicate that, for 3 different surgical simulations representing procedures of varying complexity, assessments by a single assessment tool is not adequate. To completely understand a resident's abilities, assessment by checklist (understanding the steps of the surgery), global rating scales (assessment of basic surgical skills in light of lesser or greater complexity surgeries), and pass/fail assessment (examination of adverse events) are all necessary components. TYPE OF STUDY/LEVEL OF EVIDENCE Therapeutic II.
Collapse
Affiliation(s)
- Ann VanHeest
- Department of Orthopaedic Surgery and Department of Surgery, University of Minnesota, Minneapolis, MN, USA.
| | | | | | | | | | | |
Collapse
|
27
|
Hawkins SC, Osborne A, Schofield SJ, Pournaras DJ, Chester JF. Improving the accuracy of self-assessment of practical clinical skills using video feedback--the importance of including benchmarks. MEDICAL TEACHER 2012; 34:279-84. [PMID: 22455696 DOI: 10.3109/0142159x.2012.658897] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2023]
Abstract
INTRODUCTION Isolated video recording has not been demonstrated to improve self-assessment accuracy. This study examines if the inclusion of a defined standard benchmark performance in association with video feedback of a student's own performance improves the accuracy of student self-assessment of clinical skills. METHODS Final year medical students were video recorded performing a standardised suturing task in a simulated environment. After the exercise, the students self-assessed their performance using global rating scales (GRSs). An identical self-assessment process was repeated following video review of their performance. Students were then shown a video-recorded 'benchmark performance', which was specifically developed for the study. This demonstrated the competency levels required to score full marks (30 points). A further self-assessment task was then completed. Students' scores were correlated against expert assessor scores. RESULTS A total of 31 final year medical students participated. Student self-assessment scores before video feedback demonstrated moderate positive correlation with expert assessor scores (r = 0.48, p < 0.01) with no change after video feedback (r = 0.49, p < 0.01). After video feedback with benchmark performance demonstration, self-assessment scores demonstrated a very strong positive correlation with expert scores (r = 0.83, p < 0.0001). CONCLUSIONS The demonstration of a video-recorded benchmark performance in combination with video feedback may significantly improve the accuracy of students' self-assessments.
Collapse
|
28
|
Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N. The impact of nontechnical skills on technical performance in surgery: a systematic review. J Am Coll Surg 2011; 214:214-30. [PMID: 22200377 DOI: 10.1016/j.jamcollsurg.2011.10.016] [Citation(s) in RCA: 238] [Impact Index Per Article: 18.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2011] [Revised: 10/27/2011] [Accepted: 10/31/2011] [Indexed: 01/08/2023]
Abstract
BACKGROUND Failures in nontechnical and teamwork skills frequently lie at the heart of harm and near-misses in the operating room (OR). The purpose of this systematic review was to assess the impact of nontechnical skills on technical performance in surgery. STUDY DESIGN MEDLINE, EMBASE, PsycINFO databases were searched, and 2,041 articles were identified. After limits were applied, 341 articles were retrieved for evaluation. Of these, 28 articles were accepted for this review. Data were extracted from the articles regarding sample population, study design and setting, measures of nontechnical skills and technical performance, study findings, and limitations. RESULTS Of the 28 articles that met inclusion criteria, 21 articles assessed the impact of surgeons' nontechnical skills on their technical performance. The evidence suggests that receiving feedback and effectively coping with stressful events in the OR has a beneficial impact on certain aspects of technical performance. Conversely, increased levels of fatigue are associated with detriments to surgical skill. One article assessed the impact of anesthesiologists' nontechnical skills on anesthetic technical performance, finding a strong positive correlation between the 2 skill sets. Finally, 6 articles assessed the impact of multiple nontechnical skills of the entire OR team on surgical performance. A strong relationship between teamwork failure and technical error was empirically demonstrated in these studies. CONCLUSIONS Evidence suggests that certain nontechnical aspects of performance can enhance or, if lacking, contribute to deterioration of surgeons' technical performance. The precise extent of this effect remains to be elucidated.
Collapse
Affiliation(s)
- Louise Hull
- Department of Surgery and Cancer, Imperial College London, London, United Kingdom
| | | | | | | | | | | |
Collapse
|
29
|
Alvand A, Auplish S, Khan T, Gill HS, Rees JL. Identifying orthopaedic surgeons of the future. ACTA ACUST UNITED AC 2011; 93:1586-91. [DOI: 10.1302/0301-620x.93b12.27946] [Citation(s) in RCA: 40] [Impact Index Per Article: 3.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
The aim of this study was to investigate the effect of training on the arthroscopic performance of a group of medical students and to determine whether all students could be trained to competence. Thirty-three medical students with no previous experience of arthroscopy were randomised to a ‘Trained’ or an ‘Untrained’ cohort. They were required to carry out 30 episodes of two simulated arthroscopic tasks (one shoulder and one knee). The primary outcome variable was task success at each episode. Individuals achieved competence when their learning curve stabilised. The secondary outcome was technical dexterity, assessed objectively using a validated motion analysis system. Six subjects in the ‘Untrained’ cohort failed to achieve competence in the shoulder task, compared with one in the ‘Trained’ cohort. During the knee task, two subjects in each cohort failed to achieve competence. Based on the objective motion analysis parameters, the ‘Trained’ cohort performed better on the shoulder task (p < 0.05) but there was no significant difference for the knee task (p > 0.05). Although specific training improved the arthroscopic performance of novices, there were individuals who could not achieve competence despite focused training.These findings may have an impact on the selection process for trainees and influence individual career choices.
Collapse
Affiliation(s)
- A. Alvand
- Nuffield Department of Orthopaedics, Rheumatology
and Musculoskeletal Sciences, NIHR Biomedical
Research Unit, University of Oxford, Windmill
Road, Oxford OX3 7LD, UK
| | - S. Auplish
- Nuffield Department of Orthopaedics, Rheumatology
and Musculoskeletal Sciences, NIHR Biomedical
Research Unit, University of Oxford, Windmill
Road, Oxford OX3 7LD, UK
| | - T. Khan
- Nuffield Department of Orthopaedics, Rheumatology
and Musculoskeletal Sciences, NIHR Biomedical
Research Unit, University of Oxford, Windmill
Road, Oxford OX3 7LD, UK
| | - H. S. Gill
- Nuffield Department of Orthopaedics, Rheumatology
and Musculoskeletal Sciences, NIHR Biomedical
Research Unit, University of Oxford, Windmill
Road, Oxford OX3 7LD, UK
| | - J. L. Rees
- Nuffield Department of Orthopaedics, Rheumatology
and Musculoskeletal Sciences, NIHR Biomedical
Research Unit, University of Oxford, Windmill
Road, Oxford OX3 7LD, UK
| |
Collapse
|
30
|
Hanson C, Mossey P, Macluskey M. The assessment of suturing skills of dental undergraduates. EUROPEAN JOURNAL OF DENTAL EDUCATION : OFFICIAL JOURNAL OF THE ASSOCIATION FOR DENTAL EDUCATION IN EUROPE 2010; 14:113-117. [PMID: 20522111 DOI: 10.1111/j.1600-0579.2009.00600.x] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2023]
Abstract
INTRODUCTION To improve the assessment of suturing skills, we firstly developed a checklist based assessment. As an aid to the teaching of suturing skills a video was subsequently developed. MATERIALS AND METHODS To validate the checklist a cohort of 57 final year students were videoed performing the formative suturing assessment which was assessed by two examiners independently. Two cohorts of third year students took part in the formal assessment but only the second cohort of students would have access to the video. RESULTS The first cohort of 58 third year students' success rate was 93% and 1 year later 94% of 53 third year students were successful. One hundred per cent of the first cohort and 98% of the second cohort thought that the exam was a fair assessment of suturing skills. The majority of students thought that the checklist was helpful (94% and 93%). However, 62% and 55% thought that the assessment was a stressful experience. However, only 80% of the first cohort compared with 98% of the second cohort thought that they could now place sutures in intra-oral wounds. CONCLUSION Our findings suggest that checklist based assessments of suturing skills were well received by the students and improved the objectivity and transparency of the assessment process. Further work is required to determine if teaching initially on a tabletop model can be transferred to the dental clinical situation.
Collapse
Affiliation(s)
- C Hanson
- Unit of Oral Surgery and Medicine, University of Dundee Dental School, Park Place, Dundee, United Kingdom
| | | | | |
Collapse
|
31
|
Beard J, Rowley D, Bussey M, Pitts D. Workplace-based assessment: assessing technical skill throughout the continuum of surgical training. ANZ J Surg 2009; 79:148-53. [PMID: 19317780 DOI: 10.1111/j.1445-2197.2008.04832.x] [Citation(s) in RCA: 37] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
The Royal Colleges of Surgeons and Surgical Specialty Associations in the UK have introduced competence-based syllabi and curricula for surgical training. The syllabi of the Intercollegiate Surgical Curriculum Programme (ISCP) and Orthopaedic Curriculum and Assessment Programme (OCAP) define the core competencies, that is, the observable and measureable behaviours required of a surgical trainee. The curricula define when, where and how these will be assessed. Procedure-based assessment (PBA) has been adopted as the principal method of assessing surgical skills. It combines competencies specific to the procedure with generic competencies such as safe handling of instruments. It covers the entire procedure, including preoperative and postoperative planning. A global summary of the level at which the trainee performed the assessed elements of the procedure is also included. The form has been designed to be completed quickly by the assessor (clinical supervisor) and fed-back to the trainee between operations. PBA forms have been developed for all index procedures in all surgical specialties. The forms are intended to be used as frequently as possible when performing index procedures, as their primary aim is to aid learning. At the end of a training placement the aggregated PBA forms, together with the logbook, enable the Educational Supervisor and/or Programme Director to make a summary judgement about the competence of a trainee to perform index procedures to a given standard.
Collapse
|
32
|
Leong JJH, Leff DR, Das A, Aggarwal R, Reilly P, Atkinson HDE, Emery RJ, Darzi AW. Validation of orthopaedic bench models for trauma surgery. ACTA ACUST UNITED AC 2008; 90:958-65. [PMID: PMID: 18591610 DOI: 10.1302/0301-620x.90b7.20230] [Citation(s) in RCA: 55] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
The aim of this study was to validate the use of three models of fracture fixation in the assessment of technical skills. We recruited 21 subjects (six experts, seven intermediates, and eight novices) to perform three procedures: application of a dynamic compression plate on a cadaver porcine model, insertion of an unreamed tibial intramedullary nail, and application of a forearm external fixator, both on synthetic bone models. The primary outcome measures were the Objective Structural Assessment of technical skills global rating scale on video recordings of the procedures which were scored by two independent expert observers, and the hand movements of the surgeons which were analysed using the Imperial College Surgical Assessment Device. The video scores were significantly different for the three groups in all three procedures (p < 0.05), with excellent inter-rater reliability (alpha = 0.88). The novice and intermediate groups specifically were significantly different in their performance with dynamic compression plate and intramedullary nails (p < 0.05). Movement analysis distinguished between the three groups in the dynamic compression plate model, but a ceiling effect was demonstrated in the intramedullary nail and external fixator procedures, where intermediates and experts performed to comparable standards (p > 0.6). A total of 85% (18 of 21) of the subjects found the dynamic compression model and 57% (12 of 21) found all the models acceptable tools of assessment. This study has validated a low-cost, high-fidelity porcine dynamic compression plate model using video rating scores for skills assessment and movement analysis. It has also demonstrated that Synbone models for the application of and intramedullary nail and an external fixator are less sensitive and should be improved for further assessment of surgical skills in trauma. The availability of valid objective tools of assessment of surgical skills allows further studies into improving methods of training.
Collapse
|
33
|
McKinley RK, Strand J, Ward L, Gray T, Alun-Jones T, Miller H. Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review. MEDICAL EDUCATION 2008; 42:338-349. [PMID: 18338987 DOI: 10.1111/j.1365-2923.2007.02970.x] [Citation(s) in RCA: 34] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
OBJECTIVE To develop generic criteria for the global assessment of clinical procedural competence and to quantify the extent to which existing checklists allow for holistic assessment of procedural competencies. METHODS We carried out a systematic review and qualitative analysis of published clinical procedural skills assessment checklists and enumerated the contents of each. Source materials included all English-language papers published from 1990 to June 2005, identified from 18 databases, which described or referred to an assessment document for any clinical procedural skill. A pair of reviewers identified key generic themes and sub-themes through in-depth analysis of a subset of 20 checklists with iterative agreement and independent retesting of a coding framework. The resulting framework was independently applied to all checklists by pairs of reviewers checking for the emergence of new themes and sub-themes. Main outcome measures were identification of generic clinical procedural skills and the frequency of occurrence of each in the identified checklists. RESULTS We identified 7 themes ('Procedural competence', represented in 85 [97%] checklists; 'Preparation', 65 [74%]; 'Safety', 45 [51%]; 'Communication and working with the patient', 32 [36%]; 'Infection control', 28 [32%]; 'Post-procedural care', 24 [27%]; 'Team working', 13 [15%]) and 37 sub-themes, which encapsulated all identified checklists. Of the sub-themes, 2 were identified after the initial coding framework had been finalised. CONCLUSIONS It is possible to develop generic criteria for the global assessment of clinical procedural skills. A third and a half of checklists, respectively, do not enable explicit assessment of the key competencies 'Infection control' and 'Safety'. Their assessment may be inconsistent in assessments which use such checklists.
Collapse
|
34
|
Abstract
I describe an approach to predicting competence in technical skills for the purposes of resident selection. To demonstrate a predictive relationship, it is necessary to use measures that exhibit variation, reliability, and validity. There is little evidence that such measures are routinely used in the process of selecting residents. I argue that the selection of assessment instruments in the predictor domain must be guided by relevant theoretical considerations, while assessment in the surgical domain must make use of more objective and reliable instruments than is currently the practice. I present a brief summary of research on predicting operative technical competence.
Collapse
Affiliation(s)
- Stanley J Hamstra
- Department of Medical Education, University of Michigan, Ann Arbor, MI 48109-0201, USA.
| |
Collapse
|
35
|
Hislop SJ, Hsu JH, Narins CR, Gillespie BT, Jain RA, Schippert DW, Almudevar AL, Illig KA. Simulator assessment of innate endovascular aptitude versus empirically correct performance. J Vasc Surg 2006; 43:47-55. [PMID: 16414386 DOI: 10.1016/j.jvs.2005.09.035] [Citation(s) in RCA: 53] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2005] [Accepted: 09/24/2005] [Indexed: 11/20/2022]
Abstract
OBJECTIVE Prior work has established that performance on an endovascular simulator incorporating tactile feedback (haptics) correlates with previous endovascular experience and can be improved by training. This study was designed to test the ability to define and measure innate endovascular aptitude and empirically correct performance and to determine whether these are two different things. METHODS Subjects ranging in endovascular skill level from novice to expert were surveyed to determine video game experience and skill, endovascular level of training, and endovascular experience. They were then tested by using a standard protocol requiring timed advancement of a catheter and wire sequentially into each of three vessels arising from a simulated type I arch. Recorded trials were independently and blindly scored by two experienced endovascular faculty members by using a modification of a previously validated scale (Modified Reznick Scale; MRS). Summed scores were analyzed by frequency analysis and categorized as satisfactory and unsatisfactory on the basis of a clear bimodal distribution. Categorical outcome, time to task completion, and other variables were analyzed by means of linear regression, analysis of variance, and Welch modified two-sample t tests, as indicated. RESULTS A total of 61 subjects were enrolled: 42% students, 8% technicians, 19% surgeons, 13% cardiologists, and 18% radiologists. Of these, 62% were considered novices and 30% experts on the basis of previous experience; 56% of subjects worked in an endovascular-related occupation. MRS scores were highly correlated between raters (P < .0001) and showed a clear bimodal distribution, with subjects in any endovascular occupation (including technicians) scoring significantly better than all others (P < .0001). Hours of video games played per week were correlated highly with completion times (P < .001) and MRS scores (P < .001). Measures of formal training (number of endovascular cases and occupation) correlated highly with completion times (all P < .03) and MRS scores (all P < .008). In comparing completion times vs MRS scores, three groups were apparent: unskilled-inexperienced, skilled-inexperienced, and skilled-experienced, corresponding primarily to senior subjects without endovascular experience, younger subjects without endovascular experience, and formally trained endovascular physicians, respectively. Those judged intermediate in aptitude reduced times to the lowest possible level before improving their MRS scores. CONCLUSIONS Although inherently subjective, the MRS yields reproducible scores that correlate with endovascular experience and formal training. Experts and novices with extensive video game experience achieve short completion times, whereas high MRS scores are achieved only by formally trained subjects. Innate endovascular aptitude and empirically correct performance may be two separate things, and aptitude may be acquirable through (or identified by) extensive nonmedical video game experience.
Collapse
Affiliation(s)
- Sean J Hislop
- Division of Vascular Surgery, University of Rochester Medical Center, Rochester, NY 14642, USA
| | | | | | | | | | | | | | | |
Collapse
|