Development of performance and learning rate evaluation models in robot-assisted surgery using electroencephalography and eye-tracking

The existing performance evaluation methods in robot-assisted surgery (RAS) are mainly subjective, costly, and affected by shortcomings such as the inconsistency of results and dependency on the raters' opinions. The aim of this study was to develop models for an objective evaluation of performance and rate of learning RAS skills while practicing surgical simulator tasks. The electroencephalogram (EEG) and eye-tracking data were recorded from 26 subjects while performing Tubes, Suture Sponge, and Dots and Needles tasks. Performance scores were generated by the simulator program. The functional brain networks were extracted using EEG data and coherence analysis. Then these networks, along with community detection analysis, facilitated the extraction of average search information and average temporal flexibility features at 21 Brodmann areas (BA) and four band frequencies. Twelve eye-tracking features were extracted and used to develop linear random intercept models for performance evaluation and multivariate linear regression models for the evaluation of the learning rate. Results showed that subject-wise standardization of features improved the R2 of the models. Average pupil diameter and rate of saccade were associated with performance in the Tubes task (multivariate analysis; p-value = 0.01 and p-value = 0.04, respectively). Entropy of pupil diameter was associated with performance in Dots and Needles task (multivariate analysis; p-value = 0.01). Average temporal flexibility and search information in several BAs and band frequencies were associated with performance and rate of learning. The models may be used to objectify performance and learning rate evaluation in RAS once validated with a broader sample size and tasks.

Surgical skill level classification model development using EEG and eye-gaze data and machine learning algorithms

The aim of this study was to develop machine learning classification models using electroencephalogram (EEG) and eye-gaze features to predict the level of surgical expertise in robot-assisted surgery (RAS). EEG and eye-gaze data were recorded from 11 participants who performed cystectomy, hysterectomy, and nephrectomy using the da Vinci robot. Skill level was evaluated by an expert RAS surgeon using the modified Global Evaluative Assessment of Robotic Skills (GEARS) tool, and data from three subtasks were extracted to classify skill levels using three classification models-multinomial logistic regression (MLR), random forest (RF), and gradient boosting (GB). The GB algorithm was used with a combination of EEG and eye-gaze data to classify skill levels, and differences between the models were tested using two-sample t tests. The GB model using EEG features showed the best performance for blunt dissection (83% accuracy), retraction (85% accuracy), and burn dissection (81% accuracy). The combination of EEG and eye-gaze features using the GB algorithm improved the accuracy of skill level classification to 88% for blunt dissection, 93% for retraction, and 86% for burn dissection. The implementation of objective skill classification models in clinical settings may enhance the RAS surgical training process by providing objective feedback about performance to surgeons and their teachers.

Neurocognitive Correlates of Clinical Decision Making: A Pilot Study Using Electroencephalography

The development of sound clinical reasoning, while essential for optimal patient care, can be quite an elusive process. Researchers typically rely on a self-report or observational measures to study decision making, but clinicians' reasoning processes may not be apparent to themselves or outside observers. This study explored electroencephalography (EEG) to examine neurocognitive correlates of clinical decision making during a simulated American Board of Anesthesiology-style standardized oral exam. Eight novice anesthesiology residents and eight fellows who had recently passed their board exams were included in the study. Measures included EEG recordings from each participant, demographic information, self-reported cognitive load, and observed performance. To examine neurocognitive correlates of clinical decision making, power spectral density (PSD) and functional connectivity between pairs of EEG channels were analyzed. Although both groups reported similar cognitive load (p = 0.840), fellows outperformed novices based on performance scores (p < 0.001). PSD showed no significant differences between the groups. Several coherence features showed significant differences between fellows and residents, mostly related to the channels within the frontal, between the frontal and parietal, and between the frontal and temporal areas. The functional connectivity patterns found in this study could provide some clues for future hypothesis-driven studies in examining the underlying cognitive processes that lead to better clinical reasoning.

Performance and learning rate prediction models development in FLS and RAS surgical tasks using electroencephalogram and eye gaze data and machine learning

OBJECTIVE

This study explored the use of electroencephalogram (EEG) and eye gaze features, experience-related features, and machine learning to evaluate performance and learning rates in fundamentals of laparoscopic surgery (FLS) and robotic-assisted surgery (RAS).

METHODS

EEG and eye-tracking data were collected from 25 participants performing three FLS and 22 participants performing two RAS tasks. Generalized linear mixed models, using L1-penalized estimation, were developed to objectify performance evaluation using EEG and eye gaze features, and linear models were developed to objectify learning rate evaluation using these features and performance scores at the first attempt. Experience metrics were added to evaluate their role in learning robotic surgery. The differences in performance across experience levels were tested using analysis of variance.

RESULTS

EEG and eye gaze features and experience-related features were important for evaluating performance in FLS and RAS tasks with reasonable results. Residents outperformed faculty in FLS peg transfer (p value = 0.04), while faculty and residents both excelled over pre-medical students in the FLS pattern cut (p value = 0.01 and p value < 0.001, respectively). Fellows outperformed pre-medical students in FLS suturing (p value = 0.01). In RAS tasks, both faculty and fellows surpassed pre-medical students (p values for the RAS pattern cut were 0.001 for faculty and 0.003 for fellows, while for RAS tissue dissection, the p value was less than 0.001 for both groups), with residents also showing superior skills in tissue dissection (p value  = 0.03).

CONCLUSION

Findings could be used to develop training interventions for improving surgical skills and have implications for understanding motor learning and designing interventions to enhance learning outcomes.

Developing cognitive workload and performance evaluation models using functional brain network analysis

Cognition, defined as the ability to learn, remember, sustain attention, make decisions, and solve problems, is essential in daily activities and in learning new skills. The purpose of this study was to develop cognitive workload and performance evaluation models using features that were extracted from Electroencephalogram (EEG) data through functional brain network and spectral analyses. The EEG data were recorded from 124 brain areas of 26 healthy participants conducting two cognitive tasks on a robot simulator. The functional brain network and Power Spectral Density features were extracted from EEG data using coherence and spectral analyses, respectively. Participants reported their perceived cognitive workload using the SURG-TLX questionnaire after each exercise, and the simulator generated actual performance scores. The extracted features, actual performance scores, and subjectively assessed cognitive workload values were used to develop linear models for evaluating performance and cognitive workload. Furthermore, the Pearson correlation was used to find the correlation between participants' age, performance, and cognitive workload. The findings demonstrated that combined EEG features retrieved from spectral analysis and functional brain networks can be used to evaluate cognitive workload and performance. The cognitive workload in conducting only Matchboard level 3, which is more challenging than Matchboard level 2, was correlated with age (0.54, p-value = 0.01). This finding may suggest playing more challenging computer games are more helpful in identifying changes in cognitive workload caused by aging. The findings could open the door for a new era of objective evaluation and monitoring of cognitive workload and performance.

Developing surgical skill level classification model using visual metrics and a gradient boosting algorithm

Objective

Assessment of surgical skills is crucial for improving training standards and ensuring the quality of primary care. This study aimed to develop a gradient boosting classification model (GBM) to classify surgical expertise into inexperienced, competent, and experienced levels in robot-assisted surgery (RAS) using visual metrics.

Methods

Eye gaze data were recorded from 11 participants performing four subtasks; blunt dissection, retraction, cold dissection, and hot dissection using live pigs and the da Vinci robot. Eye gaze data were used to extract the visual metrics. One expert RAS surgeon evaluated each participant's performance and expertise level using the modified Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. The extracted visual metrics were used to classify surgical skill levels and to evaluate individual GEARS metrics. Analysis of Variance (ANOVA) was used to test the differences for each feature across skill levels.

Results

Classification accuracies for blunt dissection, retraction, cold dissection, and burn dissection were 95%, 96%, 96%, and 96%, respectively. The time to complete only the retraction was significantly different among the 3 skill levels (p-value = 0.04). Performance was significantly different for 3 categories of surgical skill level for all subtasks (p-values<0.01). The extracted visual metrics were strongly associated with GEARS metrics (R2>0.7 for GEARS metrics evaluation models).

Conclusions

Machine learning (ML) algorithms trained by visual metrics of RAS surgeons can classify surgical skill levels and evaluate GEARS measures. The time to complete a surgical subtask may not be considered a stand-alone factor for skill level assessment.

Utilizing deep neural networks and electroencephalogram for objective evaluation of surgeon's distraction during robot-assisted surgery

OBJECTIVE

To develop an algorithm for objective evaluation of distraction of surgeons during robot-assisted surgery (RAS).

MATERIALS AND METHODS

Electroencephalogram (EEG) of 22 medical students was recorded while performing five key tasks on the robotic surgical simulator: Instrument Control, Ball Placement, Spatial Control II, Fourth Arm Tissue Retraction, and Hands-on Surgical Training Tasks. All students completed the Surgery Task Load Index (SURG-TLX), which includes one domain for subjective assessment of distraction (scale: 1-20). Scores were divided into low (score 1-6, subjective label: 1), intermediate (score 7-12, subjective label: 2), and high distraction (score 13-20, subjective label: 3). These cut-off values were arbitrarily considered based on a verbal assessment of participants and experienced surgeons. A Deep Convolutional Neural Network (CNN) algorithm was trained utilizing EEG recordings from the medical students and used to classify their distraction levels. The accuracy of our method was determined by comparing the subjective distraction scores on SURG-TLX and the results from the proposed classification algorithm. Also, Pearson correlation was utilized to assess the relationship between performance scores (generated by the simulator) and distraction (Subjective assessment scores).

RESULTS

The proposed end-to-end model classified distraction into low, intermediate, and high with 94%, 89%, and 95% accuracy, respectively. We found a significant negative correlation (r = -0.21; p = 0.003) between performance and SURG-TLX distraction scores.

CONCLUSIONS

Herein we report, to our knowledge, the first objective method to assess and quantify distraction while performing robotic surgical tasks on the robotic simulator, which may improve patient safety. Validation in the clinical setting is required.

Surgical Hand Gesture Recognition Utilizing Electroencephalogram as Input to the Machine Learning and Network Neuroscience Algorithms

Surgical gestures detection can provide targeted, automated surgical skill assessment and feedback during surgical training for robot-assisted surgery (RAS). Several sources including surgical videos, robot tool kinematics, and an electromyogram (EMG) have been proposed to reach this goal. We aimed to extract features from electroencephalogram (EEG) data and use them in machine learning algorithms to classify robot-assisted surgical gestures. EEG was collected from five RAS surgeons with varying experience while performing 34 robot-assisted radical prostatectomies over the course of three years. Eight dominant hand and six non-dominant hand gesture types were extracted and synchronized with associated EEG data. Network neuroscience algorithms were utilized to extract functional brain network and power spectral density features. Sixty extracted features were used as input to machine learning algorithms to classify gesture types. The analysis of variance (ANOVA) F-value statistical method was used for feature selection and 10-fold cross-validation was used to validate the proposed method. The proposed feature set used in the extra trees (ET) algorithm classified eight gesture types performed by the dominant hand of five RAS surgeons with an accuracy of 90%, precision: 90%, sensitivity: 88%, and also classified six gesture types performed by the non-dominant hand with an accuracy of 93%, precision: 94%, sensitivity: 94%.

Identifying mental health status using deep neural network trained by visual metrics

Mental health is an integral part of the quality of life of cancer patients. It has been found that mental health issues, such as depression and anxiety, are more common in cancer patients. They may result in catastrophic consequences, including suicide. Therefore, monitoring mental health metrics (such as hope, anxiety, and mental well-being) is recommended. Currently, there is lack of objective method for mental health evaluation, and most of the available methods are limited to subjective face-to-face discussions between the patient and psychotherapist. In this study we introduced an objective method for mental health evaluation using a combination of convolutional neural network and long short-term memory (CNN-LSTM) algorithms learned and validated by visual metrics time-series. Data were recorded by the TobiiPro eyeglasses from 16 patients with cancer after major oncologic surgery and nine individuals without cancer while viewing18 artworks in an in-house art gallery. Pre-study and post-study questionnaires of Herth Hope Index (HHI; for evaluation of hope), anxiety State-Trait Anxiety Inventory for Adults (STAI; for evaluation of anxiety) and Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS; for evaluation of mental well-being) were completed by participants. Clinical psychotherapy and statistical suggestions for cutoff scores were used to assign an individual's mental health metrics level during each session into low (class 0), intermediate (class 1), and high (class 2) levels. Our proposed model was used to objectify evaluation and categorize HHI, STAI, and WEMWBS status of individuals. Classification accuracy of the model was 93.81%, 94.76%, and 95.00% for HHI, STAI, and WEMWBS metrics, respectively. The proposed model can be integrated into applications for home-based mental health monitoring to be used by patients after oncologic surgery to identify patients at risk.

Functional Brain States Measure Mentor-Trainee Trust during Robot-Assisted Surgery

Mutual trust is important in surgical teams, especially in robot-assisted surgery (RAS) where interaction with robot-assisted interface increases the complexity of relationships within the surgical team. However, evaluation of trust between surgeons is challenging and generally based on subjective measures. Mentor-Trainee trust was defined as assessment of mentor on trainee's performance quality and approving trainee's ability to continue performing the surgery. Here, we proposed a novel method of objectively assessing mentor-trainee trust during RAS based on patterns of brain activity of surgical mentor observing trainees. We monitored the EEG activity of a mentor surgeon while he observed procedures performed by surgical trainees and quantified the mentor's brain activity using functional and cognitive brain state features. We used methods from machine learning classification to identity key features that distinguish trustworthiness from concerning performances. Results showed that during simple surgical task, functional brain features are sufficient to classify trust. While, during more complex tasks, the addition of cognitive features could provide additional accuracy, but functional brain state features drive classification performance. These results indicate that functional brain network interactions hold information that may help objective trainee specific mentorship and aid in laying the foundation of automation in the human-robot shared control environment during RAS.

Mentor's brain functional connectivity network during robotic assisted surgery mentorship

In many complicated cognitive-motor tasks mentoring is inevitable during the learning process. Although mentors are expert in doing the task, trainee's operation might be new for a mentor. This makes mentoring a very difficult task which demands not only the knowledge and experience of a mentor, but also his/her ability to follow trainee's movements and patiently advise him/her during the operation. We hypothesize that information binding throughout the mentor's brain areas, contributed to the task, changes as the expertise level of the trainee improves from novice to intermediate and expert. This can result in the change of mentor's level of satisfaction. The brain functional connectivity network is extracted by using brain activity of a mentor during mentoring novice and intermediate surgeons, watching expert surgeon operation, and doing Urethrovesical Anasthomosis (UVA) procedure by himself. By using the extracted network, we investigate the role of modularity and neural activity efficiency in mentoring. Brain activity is measured by using a 24-channel ABM Neuro-headset with the frequency of 256 Hz. One mentor operates 26 UVA procedures and three trainees with the expertise level of novice, intermediate, and expert perform 26 UVA procedures under the supervision of mentor. Our results indicate that the modularity of functional connectivity network is higher when mentor performs the task or watches the expert operation comparing mentoring the novice and intermediate surgeons. At the end of each operation, mentor subjectively assesses the quality of operation by giving scores to NASA-TLX indexes. Performance score is used to discuss our results. The extracted significant positive correlation between performance level and modularity (r = 0.38, p - value <; 0.005) shows the increase of automaticity and decrease in neural activity cost by improving the performance.

Technical mentorship during robot-assisted surgery: a cognitive analysis

OBJECTIVE

To investigate cognitive and mental workload assessments, which may play a critical role in defining successful mentorship.

MATERIALS AND METHODS

The 'Mind Maps' project aimed at evaluating cognitive function with regard to surgeon's expertise and trainee's skills. The study included electroencephalogram (EEG) recordings of a mentor observing trainee surgeons in 20 procedures involving extended lymph node dissection (eLND) or urethrovesical anastomosis (UVA), with simultaneous assessment of trainees using the National Aeronautics and Space Administration Task Load index (NASA-TLX) questionnaire. We also compared the brain activity of the mentor during this study with his own brain activity while actually performing the same surgical steps from previous procedures populated in the 'Mind Maps' project.

RESULTS

During eLND and UVA, when the mentor thought the trainee's mental demand and effort were low based on his NASA-TLX questionnaire (not satisfied with his performance), his EEG-based mental workload increased (reflecting more concern and attention). The mentor was mentally engaged and concerned while he was engrossed in observing the surgery. This was further supported by the finding that there was no significant difference in the mental demands and workload between observing and operating for the expert surgeon.

CONCLUSIONS

This study objectively evaluated the cognitive engagement of a surgical mentor teaching technical skills during surgery. The study provides a deeper understanding of how surgical teaching actually works and opens new horizons for assessment and teaching of surgery. Further research is needed to study the feasibility of this novel concept in assessment and guidance of surgical performance.

Understanding Cognitive Performance During Robot-Assisted Surgery

OBJECTIVE

To understand cognitive function of an expert surgeon in various surgical scenarios while performing robot-assisted surgery.

MATERIALS AND METHODS

In an Internal Review Board approved study, National Aeronautics and Space Administration-Task Load Index (NASA-TLX) questionnaire with surgical field notes were simultaneously completed. A wireless electroencephalography (EEG) headset was used to monitor brain activity during all procedures. Three key portions were evaluated: lysis of adhesions, extended lymph node dissection, and urethro-vesical anastomosis (UVA). Cognitive metrics extracted were distraction, mental workload, and mental state.

RESULTS

In evaluating lysis of adhesions, mental state (EEG) was associated with better performance (NASA-TLX). Utilizing more mental resources resulted in better performance as self-reported. Outcomes of lysis were highly dependent on cognitive function and decision-making skills. In evaluating extended lymph node dissection, there was a negative correlation between distraction level (EEG) and mental demand, physical demand and effort (NASA-TLX). Similar to lysis of adhesion, utilizing more mental resources resulted in better performance (NASA-TLX). Lastly, with UVA, workload (EEG) negatively correlated with mental and temporal demand and was associated with better performance (NASA-TLX). The EEG recorded workload as seen here was a combination of both cognitive performance (finding solution) and motor workload (execution). Majority of workload was contributed by motor workload of an expert surgeon. During UVA, muscle memory and motor skills of expert are keys to completing the UVA.

CONCLUSION

Cognitive analysis shows that expert surgeons utilized different mental resources based on their need.