A marker-less technique for measuring kinematics in the operating room

BACKGROUND

Often in simulated settings, quantitative analysis of technical skill relies largely on specially tagged instruments or tracers on surgeons' hands. We investigated a novel, marker-less technique for evaluating technical skill during open operations and for differentiating tasks and surgeon experience level.

METHODS

We recorded the operative field via in-light camera for open operations. Sixteen cases yielded 138 video clips of suturing and tying tasks ≥5 seconds in duration. Video clips were categorized based on surgeon role (attending, resident) and task subtype (suturing tasks: body wall, bowel anastomosis, complex anastomosis; tying tasks: body wall, superficial tying, deep tying). We tracked a region of interest on the hand to generate kinematic data. Nested, multilevel modeling addressed the nonindependence of clips obtained from the same surgeon.

RESULTS

Interaction effects for suturing tasks were seen between role and task categories for average speed (P = .04), standard deviation of speed (P = .05), and average acceleration (P = .03). There were significant differences across task categories for standard deviation of acceleration (P = .02). Significant differences for tying tasks across task categories were observed for maximum speed (P = .02); standard deviation of speed (P = .04); and average (P = .02), maximum (P < .01), and standard deviation (P = .03) of acceleration.

CONCLUSION

We demonstrated the ability to detect kinematic differences in performance using marker-less tracking during open operative cases. Suturing task evaluation was most sensitive to differences in surgeon role and task category and may represent a scalable approach for providing quantitative feedback to surgeons about technical skill.

Evaluation of Simulated Clinical Breast Exam Motion Patterns Using Marker-Less Video Tracking

OBJECTIVE

This study investigates using marker-less video tracking to evaluate hands-on clinical skills during simulated clinical breast examinations (CBEs).

BACKGROUND

There are currently no standardized and widely accepted CBE screening techniques.

METHODS

Experienced physicians attending a national conference conducted simulated CBEs presenting different pathologies with distinct tumorous lesions. Single hand exam motion was recorded and analyzed using marker-less video tracking. Four kinematic measures were developed to describe temporal (time pressing and time searching) and spatial (area covered and distance explored) patterns.

RESULTS

Mean differences between time pressing, area covered, and distance explored varied across the simulated lesions. Exams were objectively categorized as either sporadic, localized, thorough, or efficient for both temporal and spatial categories based on spatiotemporal characteristics. The majority of trials were temporally or spatially thorough (78% and 91%), exhibiting proportionally greater time pressing and time searching (temporally thorough) and greater area probed with greater distance explored (spatially thorough). More efficient exams exhibited proportionally more time pressing with less time searching (temporally efficient) and greater area probed with less distance explored (spatially efficient). Just two (5.9 %) of the trials exhibited both high temporal and spatial efficiency.

CONCLUSIONS

Marker-less video tracking was used to discriminate different examination techniques and measure when an exam changes from general searching to specific probing. The majority of participants exhibited more thorough than efficient patterns.

APPLICATION

Marker-less video kinematic tracking may be useful for quantifying clinical skills for training and assessment.

Working volume: validity evidence for a motion-based metric of surgical efficiency

BACKGROUND

The aim of this study was to evaluate working volume as a potential assessment metric for open surgical tasks.

METHODS

Surgical attendings (n = 6), residents (n = 4), and medical students (n = 5) performed a suturing task on simulated connective tissue (foam), artery (rubber balloon), and friable tissue (tissue paper). Using a motion tracking system, effective working volume was calculated for each hand. Repeated measures analysis of variance assessed differences in working volume by experience level, dominant and/or nondominant hand, and tissue type.

RESULTS

Analysis revealed a linear relationship between experience and working volume. Attendings had the smallest working volume, and students had the largest (P = .01). The 3-way interaction of experience level, hand, and material type showed attendings and residents maintained a similar working volume for dominant and nondominant hands for all tasks. In contrast, medical students' nondominant hand covered larger working volumes for the balloon and tissue paper materials (P < .05).

CONCLUSIONS

This study provides validity evidence for the use of working volume as a metric for open surgical skills. Working volume may provide a means for assessing surgical efficiency and the operative learning curve.

A Simulator for Measuring Forces During Surgical Knots

In this study new metrics were developed for assessing the performance of surgical knots. By adding sensors to a knot tying simulator we were able to measure the forces used while performing this basic and essential skill. Data were collected for both superficial tying and deep tying of square knots using the one hand and two hands techniques. Participants used significantly more force when tying a deep knot compared to a superficial knot (3.79N and 1.6N respectively). Different patterns for upward and downward forces were identified and showed that although most of the time upward forces are used (72% of the time), the downward forces are just as large. These data can be crucial for improving the safeness of knot tying. Combing these metrics with known metrics based on knot tensiometry and motion data may help provide feedback and objective assessment of knot tying skills.

A Multi-Layered Needle Injection Simulator

Insuring correct needle location is crucial in many medical procedures. This can be even more challenging for physicians injecting in a new location for the first time. Since they do not necessarily know how the tissue is supposed to feel, finding the correct location and correct depth can be difficult. In this study we designed a simulator for training needle injection. The simulator was fabricated to give a realistic feeling of injecting Botox® in the temporalis and the semispinalis muscles as part of migraine treatment. In addition the simulator provided real-time feedback of correct needle location. Nine residents and medical students evaluated the simulator. They made several errors that were corrected real time using the real time feedback provided. They found the simulator to be very useful and that the training significantly improved their confidence. The methods described in this study can easily be implemented for developing needle injection simulators for other anatomical locations.

Use of simulators to explore specialty recommendation for a palpable breast mass

BACKGROUND

The aim of this study was to evaluate recommendation patterns of different specialties for the work-up of a palpable breast mass using simulated scenarios and clinical breast examination models.

METHODS

Study participants were a convenience sample of physicians (n = 318) attending annual surgical, family practice, and obstetrics and gynecology (OB/GYN) conferences. Two different silicone-based breast models (superficial mass vs chest wall mass) were used to test clinical breast examination skills and recommendation patterns (imaging, tissue sampling, and follow-up).

RESULTS

Participants were more likely to recommend mammography (P < .001) and core biopsy (P < .0001) and less likely to recommend needle aspiration (P < .043) and 1-month follow-up (P < .001) for the chest wall mass compared with the superficial mass. Family practitioners were less likely to recommend ultrasound (P < .001) and obstetrics and gynecologists were less likely to recommend mammogram (P < .006) across models. Surgeons were more likely to recommend core biopsy and less likely to recommend needle aspiration across models (P < .001).

CONCLUSIONS

Recommendation patterns differed across the 2 models in line with existing practice guidelines. Additionally, differences in practice patterns between primary care and specialty providers may represent varying clinician capabilities, healthcare resources, and individual preferences. Our work shows that simulation may be used to track adherence to practice guidelines for breast masses.

Residents' perception of skill decay during dedicated research time

BACKGROUND

Surgery residents may take years away from clinical responsibilities for dedicated research time. As part of a longitudinal project, the study aim was to investigate residents' perceptions of clinical skill reduction during dedicated research time. Our hypothesis was that residents would perceive a greater potential reduction in skill during research time for procedures they were less confident in performing.

MATERIALS AND METHODS

Surgical residents engaged in dedicated research training at multiple training programs participated in four simulated procedures: urinary catheterization, subclavian central line, bowel anastomosis, and laparoscopic ventral hernia (LVH) repair. Using preprocedure and postprocedure surveys, participants rated procedures for confidence and difficulty. Residents also indicated the perceived level of skills reduction for the four procedures as a result of time in the laboratory.

RESULTS

Thirty-eight residents (55% female) completed the four clinical simulators. Participants had between 0-36 mo in a laboratory (M = 9.29 mo, standard deviation = 9.38). Preprocedure surveys noted lower confidence and higher perceived difficulty for performing the LVH repair followed by bowel anastomosis, central line insertion, and urinary catheterization (P < 0.05). Residents perceived the greatest reduction in bowel anastomosis and LVH repair skills compared with urinary catheterization and subclavian central line insertion (P < 0.001). Postprocedure surveys showed significant effects of the simulation scenarios on resident perception for urinary catheterization (P < 0.05) and LVH repair (P < 0.05).

CONCLUSIONS

Residents in this study expected greater skills decay for the procedures they had lower confidence performing and greater perceived difficulty. In addition, carefully adapted simulation scenarios had a significant effect on resident perception and may provide a mechanism for maintaining skills and keeping confidence grounded in experience.

Advanced Engineering Technology for Measuring Performance

The demand for competency-based assessments in surgical training is growing. Use of advanced engineering technology for clinical skills assessment allows for objective measures of hands-on performance. Clinical performance can be assessed in several ways via quantification of an assessee's hand movements (motion tracking), direction of visual attention (eye tracking), levels of stress (physiologic marker measurements), and location and pressure of palpation (force measurements). Innovations in video recording technology and qualitative analysis tools allow for a combination of observer- and technology-based assessments. Overall the goal is to create better assessments of surgical performance with robust validity evidence.

The use of error analysis to assess resident performance

BACKGROUND

The aim of this study was to assess validity of a human factors error assessment method for evaluating resident performance during a simulated operative procedure.

METHODS

Seven postgraduate year 4-5 residents had 30 minutes to complete a simulated laparoscopic ventral hernia (LVH) repair on day 1 of a national, advanced laparoscopic course. Faculty provided immediate feedback on operative errors and residents participated in a final product analysis of their repairs. Residents then received didactic and hands-on training regarding several advanced laparoscopic procedures during a lecture session and animate lab. On day 2, residents performed a nonequivalent LVH repair using a simulator. Three investigators reviewed and coded videos of the repairs using previously developed human error classification systems.

RESULTS

Residents committed 121 total errors on day 1 compared with 146 on day 2. One of 7 residents successfully completed the LVH repair on day 1 compared with all 7 residents on day 2 (P = .001). The majority of errors (85%) committed on day 2 were technical and occurred during the last 2 steps of the procedure. There were significant differences in error type (P ≤ .001) and level (P = .019) from day 1 to day 2. The proportion of omission errors decreased from day 1 (33%) to day 2 (14%). In addition, there were more technical and commission errors on day 2.

CONCLUSION

The error assessment tool was successful in categorizing performance errors, supporting known-groups validity evidence. Evaluating resident performance through error classification has great potential in facilitating our understanding of operative readiness.

Operative skill: quantifying surgeon's response to tissue properties

BACKGROUND

The aim of this study was to investigate how tissue characteristics influence psychomotor planning and performance during a suturing task. Our hypothesis was that participants would alter their technique based on tissue type with each subsequent stitch placed while suturing.

MATERIALS AND METHODS

Surgical attendings (n = 6), residents (n = 4), and medical students (n = 5) performed three interrupted sutures on different simulated materials as follows: foam (dense connective tissue), rubber balloons (artery), and tissue paper (friable tissue). An optical motion tracking system captured performance data from participants' bilateral hand movements. Path length and suture time were segmented by each individual stitch placed to investigate changes to psychomotor performance with subsequent stitch placements. Repeated measures analysis of variance was used to evaluate for main effects of stitch order on path length and suture time and interactions between stitch order, material, and experience.

RESULTS

When participants sutured the tissue paper, they changed their procedure time (F(4,44) = 5.14, P = 0.017) and path length (F(4,44) = 4.64, P = 0.003) in a linear fashion with the first stitch on the tissue paper having the longest procedure time and path length. Participants did not change their path lengths and procedure times when placing subsequent stitches in the foam (P = 0.910) and balloon materials (P = 0.769).

CONCLUSIONS

This study demonstrates quantifiable real-time adaptation by participants to material characteristics during a suturing task. Participants improved their motion-based performance with each subsequent stitch placement indicating changes in psychomotor planning or performance. This adaptation did not occur with the less difficult tasks. Motion capture technology is a promising method for investigating surgical performance and how surgeons adapt to operative complexity.

Idle time: an underdeveloped performance metric for assessing surgical skill

BACKGROUND

The aim of this study was to evaluate validity evidence using idle time as a performance measure in open surgical skills assessment.

METHODS

This pilot study tested psychomotor planning skills of surgical attendings (n = 6), residents (n = 4) and medical students (n = 5) during suturing tasks of varying difficulty. Performance data were collected with a motion tracking system. Participants' hand movements were analyzed for idle time, total operative time, and path length. We hypothesized that there will be shorter idle times for more experienced individuals and on the easier tasks.

RESULTS

A total of 365 idle periods were identified across all participants. Attendings had fewer idle periods during 3 specific procedure steps (P < .001). All participants had longer idle time on friable tissue (P < .005).

CONCLUSIONS

Using an experimental model, idle time was found to correlate with experience and motor planning when operating on increasingly difficult tissue types. Further work exploring idle time as a valid psychomotor measure is warranted.

Characterizing touch using pressure data and auto regressive models

Palpation plays a critical role in medical physical exams. Despite the wide range of exams, there are several reproducible and subconscious sets of maneuvers that are common to examination by palpation. Previous studies by our group demonstrated the use of manikins and pressure sensors for measuring and quantifying how physicians palpate during different physical exams. In this study we develop mathematical models that describe some of these common maneuvers. Dynamic pressure data was measured using a simplified testbed and different autoregressive models were used to describe the motion of interest. The frequency, direction and type of motion used were identified from the models. We believe these models can a provide better understanding of how humans explore objects in general and more specifically give insights to understand medical physical exams.

Use of decision-based simulations to assess resident readiness for operative independence

BACKGROUND

Recent literature has called into question resident readiness for operative independence at the end of general surgery training.

METHODS

We used a simulation-based exit examination to assess resident readiness. Six chief residents performed 3 simulated procedures: bowel anastomosis, laparoscopic ventral hernia (LVH) repair, and pancreaticojejunostomy. Faculty assessed resident performance using task-specific checklists, Objective Structured Assessment of Technical Skills (OSATS), and final product analysis.

RESULTS

Residents' individual task-specific checklist scores ranged from 25% to 100% across all 3 procedures. Mean OSATS scores ranged from 4.06 to 4.23/5.0. Residents scored significantly higher on "instrument knowledge" (mean = 4.78, standard deviation [SD] = 23) than "time and motion" (mean = 3.94, SD = .48, P = .025) and "ability to adapt to individual pathologic circumstances" (mean = 4.06, SD =.12, P = .002). Final product analysis revealed a range of errors, including incorrect technique and poor intraoperative planning.

CONCLUSIONS

Despite relatively high OSATS ratings, residents had a wide range of errors and procedure outcomes. Exit assessments using multiple evaluation metrics may improve awareness of residents' learning needs.

Quantifying technical skills during open operations using video-based motion analysis

INTRODUCTION

Objective quantification of technical operative skills in surgery remains poorly defined, although the delivery of and training in these skills is essential to the profession of surgery. Attempts to measure hand kinematics to quantify operative performance primarily have relied on electromagnetic sensors attached to the surgeon's hand or instrument. We sought to determine whether a similar motion analysis could be performed with a marker-less, video-based review, allowing for a scalable approach to performance evaluation.

METHODS

We recorded six reduction mammoplasty operations-a plastic surgery procedure in which the attending and resident surgeons operate in parallel. Segments representative of surgical tasks were identified with Multimedia Video Task Analysis software. Video digital processing was used to extract and analyze the spatiotemporal characteristics of hand movement.

RESULTS

Attending plastic surgeons appear to use their nondominant hand more than residents when cutting with the scalpel, suggesting more use of countertraction. While suturing, attendings were more ambidextrous, with smaller differences in movement between their dominant and nondominant hands than residents. Attendings also seem to have more conservation of movement when performing instrument tying than residents, as demonstrated by less nondominant hand displacement. These observations were consistent within procedures and between the different attending plastic surgeons evaluated in this fashion.

CONCLUSION

Video motion analysis can be used to provide objective measurement of technical skills without the need for sensors or markers. Such data could be valuable in better understanding the acquisition and degradation of operative skills, providing enhanced feedback to shorten the learning curve.

Use of cognitive task analysis to guide the development of performance-based assessments for intraoperative decision making

BACKGROUND

There is a paucity of performance-based assessments that focus on intraoperative decision making. The purpose of this article is to review the performance outcomes and usefulness of two performance-based assessments that were developed using cognitive task analysis (CTA) frameworks.

METHODS

Assessment-A used CTA to create a "think aloud" oral examination that was administered while junior residents (PGY 1-2's, N = 69) performed a porcine-based laparoscopic cholecystectomy. Assessment-B used CTA to create a simulation-based, formative assessment of senior residents' (PGY 4-5's, N = 29) decision making during a laparoscopic ventral hernia repair. In addition to survey-based assessments of usefulness, a multiconstruct evaluation was performed using eight variables.

RESULTS

When comparing performance outcomes, both approaches revealed major deficiencies in residents' intraoperative decision-making skills. Multiconstruct evaluation of the two CTA approaches revealed assessment method advantages for five of the eight evaluation areas: (1) Cognitive Complexity, (2) Content Quality, (3) Content Coverage, (4) Meaningfulness, and (5) Transfer and Generalizability.

CONCLUSIONS

The two CTA performance assessments were useful in identifying significant training needs. While there are pros and cons to each approach, the results serve as a useful blueprint for program directors seeking to develop performance-based assessments for intraoperative decision making.

Application of national testing standards to simulation-based assessments of clinical palpation skills

With the advent of simulation technology, several types of data acquisition methods have been used to capture hands-on clinical performance. Motion sensors, pressure sensors, and tool-tip interaction software are a few of the broad categories of approaches that have been used in simulation-based assessments. The purpose of this article is to present a focused review of 3 sensor-enabled simulations that are currently being used for patient-centered assessments of clinical palpation skills. The first part of this article provides a review of technology components, capabilities, and metrics. The second part provides a detailed discussion regarding validity evidence and implications using the Standards for Educational and Psychological Testing as an organizational and evaluative framework. Special considerations are given to content domain and creation of clinical scenarios from a developer's perspective. The broader relationship of this work to the science of touch is also considered.

A comparative assessment and gap analysis of commonly used team rating scales

BACKGROUND

The purpose of this article was to conduct a gap analysis of important team constructs that may be absent in widely used team assessments.

METHODS AND MATERIALS

Two assessment tools with known validity evidence (1) Non-Technical Skills for Surgeons (NOTSS) and (2) the Cannon-Bowers Scale were used to evaluate 11 teams of surgical residents (n = 33) performing simulated laparoscopic hernia repairs. Faculty raters' scores were used to compare the surveys and assess validity and reliability. Raters' detailed observation notes were used to indicate important behavioral constructs that were missing from the team rating scales.

RESULTS

When assessing inter-item correlations (reliability) four of five NOTSS' scale items had significant correlations (r = 0.9-1.0, P < 0.05) with the Cannon-Bowers items. While the correlations were only noted for three of six Cannon-Bowers items, in each instance the same four of five NOTSS items correlated with the three Cannon-Bowers items, thus providing further validity evidence for both scales. When evaluating the gap, key emerging themes included the need to focus on critical team errors, individual team member contributions, task performance, and overall team performance. These gaps, plus items from the NOTSS and Cannon-Bowers scales, were incorporated into a new rating scale.

CONCLUSIONS

Despite continued evidence of validity and reliability, there were several behavioral constructs that were not represented when using the NOTSS and Cannon-Bowers scales. Critical team errors, individual team member contributions, task performance, and overall team performance appear important in our ability to understand teams and teamwork.

Multimodality approach to classifying hand utilization for the clinical breast examination

The clinical breast examination (CBE) is performed to detect breast pathology. However, little is known regarding clinical technique and how it relates to diagnostic accuracy. We sought to quantify breast examination search patterns and hand utilization with a new data collection and analysis system. Participants performed the CBE while the sensor mapping and video camera system collected performance data. From this data, algorithms were developed that measured the number of hands used during the exam and active examination time. This system is a feasible and reliable method to collect new information on CBE techniques.

Utilizing a novel tool for the comprehensive assessment of resident operative performance

PURPOSE

A mechanism for more effective and comprehensive assessment of surgical residents' performance in the operating room (OR) is needed, especially in light of the new requirements issued by the American Board of Surgery. Furthermore, there is an increased awareness that assessments need to be more meaningful by including not only procedure-specific and general technical skills, but also nontechnical skills (NOTECHS), such as teamwork and communication skills. Our aims were to develop a methodology and create a tool that comprehensively assesses residents' operative performance.

METHODS

A procedure-specific technical skill assessment for laparoscopic colon resections was created through use of task analysis. Components of previously validated tools were added to broaden the assessment to include general technical skills and NOTECHS. Our instrument was then piloted in the OR to measure face and content validity through an iterative process with faculty evaluators. Once the tool was finalized, postgraduate 3 (PG3) and PG5 residents on a 2-month long rotation were assessed by 1 of 4 colorectal surgeons immediately after completing a case together. Construct validity was measured by evaluating the difference in scores between PG3 and PG5 residents' performance as well as the change in scores over the course of the rotation.

RESULTS

Sixty-three assessments were performed. All evaluations were completed within 48 hours of the operation. There was a statistically significant difference between the PG3 and PG5 scores on procedure-specific performance, general technical skills, NOTECHS, and overall performance. Over the course of the rotation, a statistically significant improvement was found in residents' scores on the procedure-specific portion of the assessment but not on the general surgical skills or NOTECHS.

CONCLUSION

This is a feasible, valid, and reliable assessment tool for the comprehensive evaluation of resident performance in the OR. We plan to use this tool to assess resident operative skill development and to improve direct resident feedback.