The Relationship of Technical Skills and Cognitive Workload to Errors During Robotic Surgical Exercises

Surgical Sabermetrics: A Scoping Review of Technology-enhanced Assessment of Nontechnical Skills in the Operating Room

OBJECTIVE

To evaluate the current evidence for surgical sabermetrics: digital methods of assessing surgical nontechnical skills and investigate the implications for enhancing surgical performance.

BACKGROUND

Surgeons need high-quality, objective, and timely feedback to optimize performance and patient safety. Digital tools to assess nontechnical skills have the potential to reduce human bias and aid scalability. However, we do not fully understand which of the myriad of digital metrics of performance assessment have efficacy for surgeons.

METHODS

A systematic review was conducted by searching PubMed, EMBASE, CINAHL, and PSYCINFO databases following PRISMA-ScR guidelines. MeSH terms and keywords included "Assessment," "Surgeons," and "Technology". Eligible studies included a digital assessment of nontechnical skills for surgeons, residents, and/or medical students within an operative context.

RESULTS

From 19,229 articles screened, 81 articles met the inclusion criteria. The studies varied in surgical specialties, settings, and outcome measurements. A total of 122 distinct objective, digital metrics were utilized. Studies digitally measured at least 1 category of surgical nontechnical skill using a single (n=54) or multiple objective measures (n=27). The majority of studies utilized simulation (n=48) over live operative settings (n=32). Surgical Sabermetrics has been demonstrated to be beneficial in measuring cognitive load (n=57), situation awareness (n=24), communication (n=3), teamwork (n=13), and leadership (n=2). No studies measured intraoperative decision-making.

CONCLUSIONS

The literature detailing the intersection between surgical data science and operative nontechnical skills is diverse and growing rapidly. Surgical Sabermetrics may provide a promising modifiable technique to achieve desirable outcomes for both the surgeon and the patient. This study identifies a diverse array of measurements possible with sensor devices and highlights research gaps, including the need for objective assessment of decision-making. Future studies may advance the integration of physiological sensors to provide a holistic assessment of surgical performance.

Cognitive ergonomics and robotic surgery

Cognitive ergonomics refer to mental resources and is associated with memory, sensory motor response, and perception. Cognitive workload (CWL) involves use of working memory (mental strain and effort) to complete a task. The three types of cognitive loads have been divided into intrinsic (dependent on complexity and expertise), extraneous (the presentation of tasks) and germane (the learning process) components. The effect of robotic surgery on CWL is complex because the postural, visualisation, and manipulation ergonomic benefits for the surgeon may be offset by the disadvantages associated with team separation and reduced situation awareness. Physical fatigue and workflow disruptions have a negative impact on CWL. Intraoperative CWL can be measured subjectively post hoc with the use of self-reported instruments or objectively with real-time physiological response metrics. Cognitive training can play a crucial role in the process of skill acquisition during the three stages of motor learning: from cognitive to integrative and then to autonomous. Mentorship, technical practice and watching videos are the most common traditional cognitive training methods in surgery. Cognitive training can also occur with computer-based cognitive simulation, mental rehearsal, and cognitive task analysis. Assessment of cognitive skills may offer a more effective way to differentiate robotic expertise level than automated performance (tool-based) metrics.

Assessing the efficacy of dissection gestures in robotic surgery

Our group previously defined a dissection gesture classification system that deconstructs robotic tissue dissection into its most elemental yet meaningful movements. The purpose of this study was to expand upon this framework by adding an assessment of gesture efficacy (ineffective, effective, or erroneous) and analyze dissection patterns between groups of surgeons of varying experience. We defined three possible gesture efficacies as ineffective (no meaningful effect on the tissue), effective (intended effect on the tissue), and erroneous (unintended disruption of the tissue). Novices (0 prior robotic cases), intermediates (1-99 cases), and experts (≥ 100 cases) completed a robotic dissection task in a dry-lab training environment. Video recordings were reviewed to classify each gesture and determine its efficacy, then dissection patterns between groups were analyzed. 23 participants completed the task, with 9 novices, 8 intermediates with median caseload 60 (IQR 41-80), and 6 experts with median caseload 525 (IQR 413-900). For gesture selection, we found increasing experience associated with increasing proportion of overall dissection gestures (p = 0.009) and decreasing proportion of retraction gestures (p = 0.009). For gesture efficacy, novices performed the greatest proportion of ineffective gestures (9.8%, p < 0.001), intermediates commit the greatest proportion of erroneous gestures (26.8%, p < 0.001), and the three groups performed similar proportions of overall effective gestures, though experts performed the greatest proportion of effective retraction gestures (85.6%, p < 0.001). Between groups of experience, we found significant differences in gesture selection and gesture efficacy. These relationships may provide insight into further improving surgical training.