Objective assessment of technical performance

Development of a human machine interface for robotically assisted surgery optimized for laparoscopic workflows

INTRODUCTION

In robotic-assisted surgery (RAS), the input device is the primary site for the flow of information between the user and the robot. Most RAS systems remove the surgeon's console from the sterile surgical site. Beneficial for performing lengthy procedures with complex systems, this ultimately lacks the flexibility that comes with the surgeon being able to remain at the sterile site.

METHODS

A prototype of an input device for RAS is constructed. The focus lies on intuitive control for surgeons and a seamless integration into the surgical workflow within the sterile environment. The kinematic design is translated from the kinematics of laparoscopic surgery. The input device uses three degrees of freedom from a flexible instrument as input. The prototype's performance is compared to that of a commercially available device in an evaluation. Metrics are used to evaluate the surgeons' performance with the respective input device in a virtual environment implemented for the evaluation.

RESULTS

The evaluation of the two input devices shows statistically significant differences in the performance metrics. With the proposed prototype, the surgeons perform the tasks faster, more precisely, and with fewer errors.

CONCLUSION

The prototype is an efficient and intuitive input device for surgeons with laparoscopic experience. The placement in the sterile working area allows for seamless integration into the surgical workflow and can potentially enable new robotic approaches.

Correlating Personal Resourcefulness and Psychomotor Skills: An Analysis of Stress, Visual Attention and Technical Metrics

Modern surgical education is focused on making use of the available technologies in order to train and assess surgical skill acquisition. Innovative technologies for the automatic, objective assessment of nontechnical skills are currently under research. The main aim of this study is to determine whether personal resourcefulness can be assessed by monitoring parameters that are related to stress and visual attention and whether there is a relation between these and psychomotor skills in surgical education. For this purpose, we implemented an application in order to monitor the electrocardiogram (ECG), galvanic skin response (GSR), gaze and performance of surgeons-in-training while performing a laparoscopic box-trainer task so as to obtain technical and personal resourcefulness' metrics. Eight surgeons (6 nonexperts and 2 experts) completed the experiment. A total of 22 metrics were calculated (7 technical and 15 related to personal resourcefulness) per subject. The average values of these metrics in the presence of stressors were compared with those in their absence and depending on the participants' expertise. The results show that both the mean normalized GSR signal and average surgical instrument's acceleration change significantly when stressors are present. Additionally, the GSR and acceleration were found to be correlated, which indicates that there is a relation between psychomotor skills and personal resourcefulness.

Quality Assurance and Improvement Project in Echocardiography Laboratory: The Pivotal Importance of Organizational and Managerial Processes

Echocardiography plays a vital role in the diagnosis and management of cardiovascular conditions. Echocardiography use is progressively increasing nowadays, and this is correlated to the evolving echo indications, to the relatively new available echocardiography modes (tissue Doppler imaging, speckle tracking imaging, three-dimensional mode, etc.) and modalities (transthoracic, transesophageal, and intracardiac) along with the various available clinical approaches (point of care echo, portable echo, etc.). Quality assurance in echocardiography is correlated to appropriate use criteria, adequate equipment, standardization of performance and reporting, along with timely storage and archiving. Quality improvement plan must target strategic planning, with metrics and timeline for assessment and re-assessment of results. Improvement project aims to ensure and enhance conformity with appropriate use criteria and standardization, timely completion of exams and reports, detection of discrepancies, and continuous improvement of knowledge and skills. Strategic planning is essential in this context in order to develop organizational and managerial processes, with regular auditing for a highly professional and advanced level of echocardiography, while ensuring teamwork and standards of ethical values.

Simulation-based VATS resection of the five lung lobes: a technical skills test

BACKGROUND

Video-Assisted Thoracoscopic Surgery (VATS) lobectomy is an advanced procedure and to maximize patient safety it is important to ensure the competency of thoracic surgeons before performing the procedure. The objective of this study was to investigate validity evidence for a virtual reality simulator-based test including multiple lobes of the lungs.

METHOD

VATS experts from the department of Cardiothoracic Surgery at Rigshospitalet, Copenhagen, Denmark, worked with Surgical Science (Gothenburg, Sweden) to develop VATS lobectomy modules for the LapSim^® virtual reality simulator covering all five lobes of the lungs. Participants with varying experience in VATS were recruited and classified as either novice, intermediate, or experienced surgeons. Each participant performed VATS lobectomy on the simulator for three different randomly chosen lobes. Nine predefined simulator metrics were automatically recorded on the simulator.

RESULTS

Twenty-two novice, ten intermediate, and nine experienced surgeons performed the test resulting in a total of 123 lobectomies. Analysis of Variances (ANOVA) found significant differences between the three groups for parameters: blood loss (p < 0.001), procedure time (p < 0.001), and total instrument path length (p = 0.03). These three metrics demonstrated high internal consistency and significant test-retest reliability was found between each of them. Relevant pass/fail levels were established for each of the three metrics, 541 ml, 30 min, and 71 m, respectively.

CONCLUSION

This study provides validity evidence for a simulator-based test of VATS lobectomy competence including multiple lobes of the lungs. The test can be used to ensure basic competence at the end of a simulation-based training program for thoracic surgery trainees.

Utilization of Flexible-Wearable Sensors to Describe the Kinematics of Surgical Proficiency

BACKGROUND

Traditional assessment (e.g., checklists, videotaping) for surgical proficiency may lead to subjectivity and does not predict performance in the clinical setting. Hand motion analysis is evolving as an objective tool for grading technical dexterity; however, most devices accompany with technical limitations or discomfort. We purpose the use of flexible wearable sensors to evaluate the kinematics of surgical proficiency.

METHODS

Surgeons were recruited and performed a vascular anastomosis task in a single institution. A modified objective structured assessment of technical skills (mOSATS) was used for technical qualification. Flexible wearable sensors (BioStamp RCTM, mc10 Inc., Lexington, MA) were placed on the dorsum of the dominant hand (DH) and nondominant hand (nDH) to measure kinematic parameters: path length (T_path), mean (V_mean) and peak (V_peak) velocity, number of hand movements (N_move), ratio of DH to nDH movements (rMov), and time of task (tTask) and further compared with the mOSATS score.

RESULTS

Participants were categorized as experts (n = 12) and novices (n = 8) based on a cutoff mean mOSATS score. Significant differences for tTask (P = 0.02), rMov (P = 0.07), DH T_path (P = 0.04), V_mean (P = 0.07), V_peak (P = 0.04), and nDH N_move (P = 0.02) were in favor of the experts. Overall, mOSATS had significant correlation with tTask (r = -0.69, P = 0.001), N_move of DH (r = -0.44, P = 0.047) and nDH (r = -0.66, P = 0.001), and rMov (r = 0.52, P = 0.017).

CONCLUSIONS

Hand motion analysis evaluated by flexible wearable sensors is feasible and informative. Experts utilize coordinated two-handed motion, whereas novices perform one-handed tasks in a hastily jerky manner. These tendencies create opportunity for improvement in surgical proficiency among trainees.

Simulation in surgical education

Simulation is becoming an important tool in surgical education. Surgical faculty have been forced to modify how they teach technical skills. Instead of a complete reliance on teaching in the operating room, a structured curriculum and dedicated time in the simulation center are being used in many centers. Some of the advantages of this approach include the ability to learn and practice new procedures in a safe and nurturing environment. The disadvantages include the significant cost of virtual reality simulators and the competition, between various training programs, to gain access to simulation.

Virtual reality improves the accuracy of simulated preoperative planning in temporal bones: a feasibility and validation study

PURPOSE

Consumer-grade virtual reality (VR) has recently enabled various medical applications, but more evidence supporting their validity is needed. We investigated the accuracy of simulated surgical planning in a VR environment (VR) with temporal bones and compared it to conventional cross-sectional image viewing in picture archiving and communication system (PACS) interface.

METHODS

Five experienced otologic surgeons measured significant anatomic structures and fiducials on five fresh-frozen cadaveric temporal bones in VR and cross-sectional viewing. Primary image data were acquired by computed tomography. In total, 275 anatomical landmark measurements and 250 measurements of the distance between fiducials were obtained with both methods. Distance measurements between the fiducials were confirmed by physical measurement obtained by Vernier caliper. The experts evaluated the subjective validity of both methods on a 5-point Likert scale qualitative survey.

RESULTS

A strong correlation based on intraclass coefficient was found between the methods on both the anatomical (r > 0.900) and fiducial measurements (r > 0.916). Two-tailed paired t-test and Bland-Altman plots demonstrated high equivalences between the VR and cross-sectional viewing with mean differences of 1.9% (p = 0.396) and 0.472 mm (p = 0.065) for anatomical and fiducial measurements, respectively. Gross measurement errors due to the misidentification of fiducials occurred more frequently in the cross-sectional viewing. The mean face and content validity rating for VR were significantly better compared to cross-sectional viewing (total mean score 4.11 vs 3.39, p < 0.001).

CONCLUSION

Our study supports good accuracy and reliability of VR environment for simulated surgical planning in temporal bones compared to conventional cross-sectional visualization.

Pediatric Surgery Simulation-Based Training for the General Surgery Resident

BACKGROUND

Surgical simulation-based training (SBT) can increase resident confidence and improve performance. SBT in pediatric surgery is in its infancy and often geared toward training pediatric surgery fellows. Since case volume for various pediatric surgery-specific procedures can be low based on the rarity of the pathology involved and the level of care provided by the institution, our aim was to create a pediatric surgery simulation-based curriculum for general surgery residents to address this need.

MATERIALS AND METHODS

We performed an institutional needs assessment consisting of 4 pediatric surgeons' and 28 general surgery residents' confidence in resident ability to independently perform pediatric surgery-specific tasks and procedures using a Likert-scaled survey. These included the placement of a silastic silo for gastroschisis, a percutaneous drain for perforated necrotizing enterocolitis, and completion of a laparoscopic pyloromyotomy for pyloric stenosis. Models simulating these pathologies and curriculum for performing each procedure were generated.

RESULTS

We successfully created a model and SBT curriculum to teach general surgery residents how to place a silastic silo for patients with gastroschisis, a percutaneous drain for patients with perforated necrotizing enterocolitis, and how to complete a laparoscopic pyloromyotomy for patients with pyloric stenosis. These were deemed high fidelity models based on a survey of our pediatric surgeons.

CONCLUSIONS

We created a pediatric surgery SBT curriculum for general surgery residents, which can be used to supplement learning of various high-acuity, low-occurrence procedures. Assessment of residents and validation of scores is underway.

Developing a 3D composite training model for cranial remodeling

OBJECTIVE

Craniosynostosis correction, including cranial vault remodeling, fronto-orbital advancement (FOA), and endoscopic suturectomy, requires practical experience with complex anatomy and tools. The infrequent exposure to complex neurosurgical procedures such as these during residency limits extraoperative training. Lack of cadaveric teaching tools given the pediatric nature of synostosis compounds this challenge. The authors sought to create lifelike 3D printed models based on actual cases of craniosynostosis in infants and incorporate them into a practical course for endoscopic and open correction. The authors hypothesized that this training tool would increase extraoperative facility and familiarity with cranial vault reconstruction to better prepare surgeons for in vivo procedures.

METHODS

The authors utilized representative craniosynostosis patient scans to create 3D printed models of the calvaria, soft tissues, and cranial contents. Two annual courses implementing these models were held, and surveys were completed by participants (n = 18, 5 attending physicians, 4 fellows, 9 residents) on the day of the course. These participants were surveyed during the course and 1 year later to assess the impact of this training tool. A comparable cohort of trainees who did not participate in the course (n = 11) was also surveyed at the time of the 1-year follow-up to assess their preparation and confidence with performing craniosynostosis surgeries.

RESULTS

An iterative process using multiple materials and the various printing parameters was used to create representative models. Participants performed all major surgical steps, and we quantified the fidelity and utility of the model through surveys. All attendees reported that the model was a valuable training tool for open reconstruction (n = 18/18 [100%]) and endoscopic suturectomy (n = 17/18 [94%]). In the first year, 83% of course participants (n = 14/17) agreed or strongly agreed that the skin and bone materials were realistic and appropriately detailed; the second year, 100% (n = 16/16) agreed or strongly agreed that the skin material was realistic and appropriately detailed, and 88% (n = 14/16) agreed or strongly agreed that the bone material was realistic and appropriately detailed. All participants responded that they would use the models for their own personal training and the training of residents and fellows in their programs.

CONCLUSIONS

The authors have developed realistic 3D printed models of craniosynostosis including soft tissues that allow for surgical practice simulation. The use of these models in surgical simulation provides a level of preparedness that exceeds what currently exists through traditional resident training experience. Employing practical modules using such models as part of a standardized resident curriculum is a logical evolution in neurosurgical education and training.

Evaluation methods and impact of simulation-based training in pediatric surgery: a systematic review

PURPOSE

The aim of this study was to identify (1) the type of skill evaluation methods and (2) how the effect of training was evaluated in simulation-based training (SBT) in pediatric surgery.

METHODS

Databases of PubMed, Cochrane Library, and Web of Science were searched for articles published from January 2000 to January 2017. Search concepts of Medical Subject Heading terms were "surgery," "pediatrics," "simulation," and "training, evaluation."

RESULTS

Of 5858 publications identified, 43 were included. Twenty papers described simulators as assessment tools used to evaluate technical skills. Reviewers differentiated between experts and trainees using a scoring system (45%) and/or a checklist (25%). Simulators as training tools were described in 23 papers. While the training's effectiveness was measured using performance assessment scales (52%) and/or surveys (43%), no study investigated the improvement of the clinical outcomes after SBT.

CONCLUSION

Scoring, time, and motion analysis methods were used for the evaluation of basic techniques of laparoscopic skills. Only a few SBT in pediatric surgery have definite goals with clinical effect. Future research needs to demonstrate the educational effect of simulators as assessment or training tools on SBT in pediatric surgery.

Moving Toward Milestone-Based Assessment in Osteopathic Manipulative Medicine

Osteopathic medicine is continuing to move toward competency-based education at undergraduate and graduate levels. Competencies and Entrustable Professional Activities (EPAs) have been implemented to guide educators on the skills and abilities that osteopathic medical students and residents should be able to perform as physicians. Unfortunately, many of these skills have not been well described, and the threshold of “competence” or “entrustability” for each of these tasks remains elusive. The author presents an approach to measuring competence in the domain of osteopathic manipulative medicine using a milestone rubric to assess skills related to osteopathic screening, diagnosis, technique, and explanation. This rubric can be applied to all levels of osteopathic training and across many diagnostic and treatment modalities. Clearly defining and assessing the individual skills composing competence in osteopathic manipulative medicine will be increasingly important as medical education continues to evolve and modernize.

Surgical gesture classification using Dynamic Time Warping and affine velocity

Minimally Invasive Surgery (MIS) has become widespread as an important surgical technique due to its advantages related to pain relief and short recovery time periods. However, this approach implies the acquisition of special surgical skills, which represents a challenge in the objective assessment of surgical gestures. In this way, several studies shown that kinematics and kinetic analysis of hand movement is a valuable assessment tool of basic surgical skills in MIS. In addition, recent researches proved that human motion performed during surgery can be described as a sequence of constant affine velocity movements. In this paper, we present a novel method to classify gestures based on an affine velocity analysis of 3D motion and an implementation of the Dynamic Time Warping algorithm. In particular, affine velocity calculation correlates kinematics and geometrical variables such as curvature, torsion, and euclidean velocity, reducing the dimension of the conventional 3D problem. In this way, using the simplicity of dynamic time warping algorithm allows us to perform an accurate classification, easier to implement and understand. Experimental validation of the algorithm is presented based on the position and orientation data of a laparoscope instrument, determiMinimally Invasive Surgery (MIS) has become widespread as an important surgical technique due to its advantages related to pain relief and short recovery time periods. However, this approach implies the acquisition of special surgical skills, which represents a challenge in the objective assessment of surgical gestures. In this way, several studies shown that kinematics and kinetic analysis of hand movement is a valuable assessment tool of basic surgical skills in MIS. In addition, recent researches proved that human motion performed during surgery can be described as a sequence of constant affine velocity movements. In this paper, we present a novel method to classify gestures based on an affine velocity analysis of 3D motion and an implementation of the Dynamic Time Warping algorithm. In particular, affine velocity calculation correlates kinematics and geometrical variables such as curvature, torsion, and euclidean velocity, reducing the dimension of the conventional 3D problem. In this way, using the simplicity of dynamic time warping algorithm allows us to perform an accurate classification, easier to implement and understand. Experimental validation of the algorithm is presented based on the position and orientation data of a laparoscope instrument, determined by six cameras. Results show the advantages of the proposed method compared to conventional Multidimensional Dynamic Time Warping to classify surgical gestures in MIS.ned by six cameras. Results show the advantages of the proposed method compared to conventional Multidimensional Dynamic Time Warping to classify surgical gestures in MIS.

Skill Acquisition, Assessment, and Simulation in Minimal Access Surgery: An Evolution of Technical Training in Surgery

Diminishing resources and expanding technologies, such as minimal access surgery, have complicated the acquisition and assessment of technical skills in surgical training programs. However, these challenges have been met with both innovation and an evolution in our understanding of how learners develop technical competence and how to better measure it. As these skills continue to grow in breadth and complexity, so too must the surgical education systems’ ability. This literature review examines and describes the pressures placed on surgical education programs and the development of methods to ameliorate them with a focus on surgical simulation.

A virtual pointer to support the adoption of professional vision in laparoscopic training

PURPOSE

To assess a virtual pointer in supporting surgical trainees' development of professional vision in laparoscopic surgery.

METHODS

We developed a virtual pointing and telestration system utilizing the Microsoft Kinect movement sensor as an overlay for any imagine system. Training with the application was compared to a standard condition, i.e., verbal instruction with un-mediated gestures, in a laparoscopic training environment. Seven trainees performed four simulated laparoscopic tasks guided by an experienced surgeon as the trainer. Trainee performance was subjectively assessed by the trainee and trainer, and objectively measured by number of errors, time to task completion, and economy of movement.

RESULTS

No significant differences in errors and time to task completion were obtained between virtual pointer and standard conditions. Economy of movement in the non-dominant hand was significantly improved when using virtual pointer ([Formula: see text]). The trainers perceived a significant improvement in trainee performance in virtual pointer condition ([Formula: see text]), while the trainees perceived no difference. The trainers' perception of economy of movement was similar between the two conditions in the initial three runs and became significantly improved in virtual pointer condition in the fourth run ([Formula: see text]).

CONCLUSIONS

Results show that the virtual pointer system improves the trainer's perception of trainee's performance and this is reflected in the objective performance measures in the third and fourth training runs. The benefit of a virtual pointing and telestration system may be perceived by the trainers early on in training, but this is not evident in objective trainee performance until further mastery has been attained. In addition, the performance improvement of economy of motion specifically shows that the virtual pointer improves the adoption of professional vision- improved ability to see and use laparoscopic video results in more direct instrument movement.

The measurement of collaboration within healthcare settings: a systematic review of measurement properties of instruments

BACKGROUND

There is a growing call by consumers and governments for healthcare to adopt systems and approaches to care to improve patient safety. Collaboration within healthcare settings is an important factor for improving systems of care. By using validated measurement instruments a standardized approach to assessing collaboration is possible, otherwise it is only an assumption that collaboration is occurring in any healthcare setting.

OBJECTIVES

The objective of this review was to evaluate and compare measurement properties of instruments that measure collaboration within healthcare settings, specifically those which have been psychometrically tested and validated.

INCLUSION CRITERIA, TYPES OF PARTICIPANTS

Participants could be healthcare professionals, the patient or any non-professional who contributes to a patient's care, for example, family members, chaplains or orderlies. The term participant type means the designation of any one participant; for example 'nurse', 'social worker' or 'administrator'. More than two participant types was mandatory.

TYPES OF INTERVENTION(S)/PHENOMENA OF INTEREST

The focus of this review was the validity of tools used to measure collaboration within healthcare settings.

TYPES OF STUDIES

The types of studies considered for inclusion were validation studies, but quantitative study designs such as randomized controlled trials, controlled trials and case studies were also eligible for inclusion. Studies that focused on Interprofessional Education, were published as an abstract only, contained patient self-reporting only or were not about care delivery were excluded.

OUTCOMES

The outcome of interest was validation and interpretability of the instrument being assessed and included content validity, construct validity and reliability. Interpretability is characterized by statistics such as mean and standard deviation which can be translated to a qualitative meaning.

SEARCH STRATEGY

The search strategy aimed to find both published and unpublished studies. A three-step search strategy was utilized in this review. The databases searched included PubMed, CINAHL, Embase, Cochrane Central Register of Controlled Trials, Emerald Fulltext, MD Consult Australia, PsycARTICLES, Psychology and Behavioural Sciences Collection, PsycINFO, Informit Health Databases, Scopus, UpToDate and Web of Science. The search for unpublished studies included EThOS (Electronic Thesis Online Service), Index to Theses and ProQuest- Dissertations and Theses.

METHODOLOGICAL QUALITY

The assessment of methodological quality of the included studies was undertaken using the COSMIN checklist which is a validated tool that assesses the process of design and validation of healthcare measurement instruments.

DATA COLLECTION

An Excel spreadsheet version of COSMIN was developed for data collection which included a worksheet for extracting participant characteristics and interpretability data.

DATA SYNTHESIS

Statistical pooling of data was not possible for this review. Therefore, the findings are presented in a narrative form including tables and figures to aid in data presentation. To make a synthesis of the assessments of methodological quality of the different studies, each instrument was rated by accounting for the number of studies performed with an instrument, the appraisal of methodological quality and the consistency of results between studies.

RESULTS

Twenty-one studies of 12 instruments were included in the review. The studies were diverse in their theoretical underpinnings, target population/setting and measurement objectives. Measurement objectives included: investigating beliefs, behaviors, attitudes, perceptions and relationships associated with collaboration; measuring collaboration between different levels of care or within a multi-rater/target group; assessing collaboration across teams; or assessing internal participation of both teams and patients.Studies produced validity or interpretability data but none of the studies assessed all validity and reliability properties. However, most of the included studies produced a factor structure or referred to prior factor analysis. A narrative synthesis of the individual study factor structures was generated consisting of nine headings: organizational settings, support structures, purpose and goals; communication; reflection on process; cooperation; coordination; role interdependence and partnership; relationships; newly created professional activities; and professional flexibility.

CONCLUSIONS

Among the many instruments that measure collaboration within healthcare settings, the quality of each instrument varies; instruments are designed for specific populations and purposes, and are validated in various settings. Selecting an instrument requires careful consideration of the qualities of each. Therefore, referring to systematic reviews of measurement properties of instruments may be helpful to clinicians or researchers in instrument selection.

IMPLICATIONS FOR PRACTICE

Systematic reviews of measurement properties of instruments are valuable in aiding in instrument selection. This systematic review may be useful in instrument selection for the measurement of collaboration within healthcare settings with a complex mix of participant types. Evaluating collaboration provides important information on the strengths and limitations of different healthcare settings and the opportunities for continuous improvement via any remedial actions initiated.

IMPLICATIONS FOR RESEARCH

Development of a tool that can be used to measure collaboration within teams of healthcare professionals and non-professionals is important for practice. The use of different statistical modelling techniques, such as Item Response Theory modelling and the translation of models into Computer Adaptive Tests, may prove useful. Measurement equivalence is an important consideration for future instrument development and validation. Further development of the COSMIN tool should include appraisal for measurement equivalence. Researchers developing and validating measurement tools should consider multi-method research designs.

Interpretation of motion analysis of laparoscopic instruments based on principal component analysis in box trainer settings

BACKGROUND

Motion analysis parameters (MAPs) have been extensively validated for assessment of minimally invasive surgical skills. However, there are discrepancies on how specific MAPs, tasks, and skills match with each other, reflecting that motion analysis cannot be generalized independently of the learning outcomes of a task. Additionally, there is a lack of knowledge on the meaning of motion analysis in terms of surgical skills, making difficult the provision of meaningful, didactic feedback. In this study, new higher significance MAPs (HSMAPs) are proposed, validated, and discussed for the assessment of technical skills in box trainers, based on principal component analysis (PCA).

METHODS

Motion analysis data were collected from 25 volunteers performing three box trainer tasks (peg grasping/PG, pattern cutting/PC, knot suturing/KS) using the EVA tracking system. PCA was applied on 10 MAPs for each task and hand. Principal components were trimmed to those accounting for an explained variance > 80% to define the HSMAPs. Individual contributions of MAPs to HSMAPs were obtained by loading analysis and varimax rotation. Construct validity of the new HSMAPs was carried out at two levels of experience based on number of surgeries.

RESULTS

Three new HSMAPs per hand were defined for PG and PC tasks, and two per hand for KS task. PG presented validity for HSMAPs related to insecurity and economy of space. PC showed validity for HSMAPs related to cutting efficacy, peripheral unawareness, and confidence. Finally, KS presented validity for HSMAPs related with economy of space and knotting security.

CONCLUSIONS

PCA-defined HSMAPs can be used for technical skills' assessment. Construct validation and expert knowledge can be combined to infer how competences are acquired in box trainer tasks. These findings can be exploited to provide residents with meaningful feedback on performance. Future works will compare the new HSMAPs with valid scoring systems such as GOALS.

A Preliminary Investigation of General and Technique-specific Assessments for the Evaluation of Laparoscopic Technical Skills

Background  Both general and technique-specific assessments of technical skill have been validated in surgical education. The purpose of this study was to assess the correlation between the objective structured assessment of technical skills (OSATS) and the global operative assessment of laparoscopic skills (GOALS) rating scales using a high-fidelity porcine laparoscopic cholecystectomy model. Methods Post-graduate year-one general surgery and urology residents (n=14) performed a live laparoscopic porcine cholecystectomy. Trained surgeons rated their performance using OSATS and GOALS assessment scales. Results Pearson's correlation coefficient between OSATS and GOALS was 0.96 for overall scores. It ranged from 0.78 - 0.89 for domains that overlapped between the two scales. Conclusion There is a very high correlation between OSATS and GOALS. This implies that they likely measure similar constructs and that either may be used for summative-type assessments of trainee skill. However, further investigation is needed to determine if technique-specific assessments may provide more useful feedback in formative evaluation.

Construction and validation of a surgical skills assessment tool for general surgery residency program

OBJECTIVE

To develop and validate an instrument for measuring the acquisition of technical skills in conducting operations of increasing difficulty for use in General Surgery Residency (GSR) programs.

METHODS

we built a surgical skills assessment tool containing 11 operations in increasing levels of difficulty. For instrument validation we used the face validity method. Through an electronic survey tool (Survey MonKey®) we sent a questionnaire to Full and Emeritus members of the Brazilian College of Surgeons--CBC--all bearers of the CBC Specialist Title.

RESULTS

Of the 307 questionnaires sent we received 100 responses. For the analysis of the data collected we used the Cronbach's alpha test. We observed that, in general, the overall alpha presented with values near or greater than 0.70, meaning good consistency to assess their points of interest.

CONCLUSION

The evaluation instrument built was validated and can be used as a method of assessment of technical skill acquisition in the General Surgery Residency programs in Brazil.

Compliance Is Contagious: Using Informatics Methods to Measure the Spread of a Documentation Standard From a Preoperative Clinic

PURPOSE

Preoperative documentation is essential to coordinated care and has the potential for standardization, which may facilitate downstream clinical management.

DESIGN

An observational pre/post standardization design was used.

METHODS

We analyzed the implementation of a preoperative documentation standardization intervention in Vanderbilt's Preoperative Evaluation Clinic (VPEC) and its impact outside VPEC. A phased intervention consisted of clinician education with monthly feedback, followed by the development of a compliance dashboard and inclusion in Ongoing Professional Performance Evaluation system by VPEC. A follow-up survey was administered to measure the impact on clinical management.

FINDINGS

Adherence to standardization was improved with the addition of electronic feedback. Implementation of this system in the preoperative clinic had significant impact outside VPEC. Trainee status was a significant predictor of adoption of the standardized format.

CONCLUSIONS

Adoption of a preoperative documentation standard in a clinic had a positive impact on standardization practices in a perioperative system.

A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery

OBJECTIVE

State-of-the-art techniques for surgical data analysis report promising results for automated skill assessment and action recognition. The contributions of many of these techniques, however, are limited to study-specific data and validation metrics, making assessment of progress across the field extremely challenging.

METHODS

In this paper, we address two major problems for surgical data analysis: First, lack of uniform-shared datasets and benchmarks, and second, lack of consistent validation processes. We address the former by presenting the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), a public dataset that we have created to support comparative research benchmarking. JIGSAWS contains synchronized video and kinematic data from multiple performances of robotic surgical tasks by operators of varying skill. We address the latter by presenting a well-documented evaluation methodology and reporting results for six techniques for automated segmentation and classification of time-series data on JIGSAWS. These techniques comprise four temporal approaches for joint segmentation and classification: hidden Markov model, sparse hidden Markov model (HMM), Markov semi-Markov conditional random field, and skip-chain conditional random field; and two feature-based ones that aim to classify fixed segments: bag of spatiotemporal features and linear dynamical systems.

RESULTS

Most methods recognize gesture activities with approximately 80% overall accuracy under both leave-one-super-trial-out and leave-one-user-out cross-validation settings.

CONCLUSION

Current methods show promising results on this shared dataset, but room for significant progress remains, particularly for consistent prediction of gesture activities across different surgeons.

SIGNIFICANCE

The results reported in this paper provide the first systematic and uniform evaluation of surgical activity recognition techniques on the benchmark database.

Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy

BACKGROUND

The societies of thoracic surgery are working to incorporate simulation and competency-based assessment into specialty training. One challenge is the development of a simulation-based test, which can be used as an assessment tool. The study objective was to establish validity evidence for a virtual reality simulator test of a video-assisted thoracoscopic surgery (VATS) lobectomy of a right upper lobe.

METHODS

Participants with varying experience in VATS lobectomy were included. They were familiarized with a virtual reality simulator (LapSim^®) and introduced to the steps of the procedure for a VATS right upper lobe lobectomy. The participants performed two VATS lobectomies on the simulator with a 5-min break between attempts. Nineteen pre-defined simulator metrics were recorded.

RESULTS

Fifty-three participants from nine different countries were included. High internal consistency was found for the metrics with Cronbach's alpha coefficient for standardized items of 0.91. Significant test-retest reliability was found for 15 of the metrics (p-values <0.05). Significant correlations between the metrics and the participants VATS lobectomy experience were identified for seven metrics (p-values <0.001), and 10 metrics showed significant differences between novices (0 VATS lobectomies performed) and experienced surgeons (>50 VATS lobectomies performed). A pass/fail level defined as approximately one standard deviation from the mean metric scores for experienced surgeons passed none of the novices (0 % false positives) and failed four of the experienced surgeons (29 % false negatives).

CONCLUSION

This study is the first to establish validity evidence for a VATS right upper lobe lobectomy virtual reality simulator test. Several simulator metrics demonstrated significant differences between novices and experienced surgeons and pass/fail criteria for the test were set with acceptable consequences. This test can be used as a first step in assessing thoracic surgery trainees' VATS lobectomy competency.

Accelerometer Measurement of Head Movement During Laparoscopic Surgery as a Tool to Evaluate Skill Development of Surgeons

OBJECTIVE

To detect and measure surgeons' head movement during laparoscopic simulator performance to determine whether expert surgeons have economy of motion in their head movement, including change of direction, compared with intermediate and novice surgeons. We investigated head movement as an objective tool for assessment of laparoscopic surgical skill and its potential use for assessing novice surgeons' progress on the learning curve.

DESIGN

After obtaining institutional review board approval, medical students, urology residents, and attending staff surgeons from an academic institution were recruited. Participants were grouped by level of experience and performed tasks on the Electronic Data Generation for Evaluation laparoscopic simulator. Surgeons wore a commercially available wireless electroencephalogram monitor as a flexible, adjustable, and lightweight headband with 7 sensors-2 forehead sensors, 2 ear sensors, and 3 reference sensors. The headband incorporates a 3-axis accelerometer enabling head movement quantification. A variance analysis was used to compare the average head movement acceleration data between each group.

SETTING

Tulane University Medical Center, New Orleans, LA, an academic medical center and the principal teaching hospital for Tulane University School of Medicine.

PARTICIPANTS

A total of following 19 participants were recruited for the study and stratified by surgical experience into novice (n = 6), intermediate (n = 9), and expert (n = 4) laparoscopy groups: 6 medical students, 9 urology residents (postgraduate years 1 to5), and 4 attending urologists, respectively.

RESULTS

Analysis of the average acceleration rate of head movement showed statistically significant differences among groups on both the vertical axis (p = 0.006) and horizontal axis (p = 0.018) in the laparoscopic suturing task. This demonstrated the ability to distinguish between experts and novice laparoscopic surgeons. The average acceleration among groups did not demonstrate statistical significance on the vertical axis (p = 0.078) and horizontal axis (p = 0.077) in the peg transfer task. This may be in response to the ease of the task. The analysis of the forward-backward axis or depth perception also showed no significant differences between groups.

CONCLUSION

Accelerometer-based motion analysis of head movement appears to be a useful tool to evaluate laparoscopic skill development of surgeons in terms of their economy of motion, and it could potentially be used for ergonomic assessment of training in the future, and progression on the learning curve.

Laparoscopic surgery skills evaluation: analysis based on accelerometers

Background and Objective:

Technical skills assessment is considered an important part of surgical training. Subjective assessment is not appropriate for training feedback, and there is now increased demand for objective assessment of surgical performance. Economy of movement has been proposed as an excellent alternative for this purpose. The investigators describe a readily available method to evaluate surgical skills through motion analysis using accelerometers in Apple's iPod Touch device.

Methods:

Two groups of individuals with different minimally invasive surgery skill levels (experts and novices) were evaluated. Each group was asked to perform a given task with an iPod Touch placed on the dominant-hand wrist. The Accelerometer Data Pro application makes it possible to obtain movement-related data detected by the accelerometers. Average acceleration and maximum acceleration for each axis (x, y, and z) were determined and compared.

Results:

The analysis of average acceleration and maximum acceleration showed statistically significant differences between groups on both the y (P = .04, P = .03) and z (P = .04, P = .04) axes. This demonstrates the ability to distinguish between experts and novices. The analysis of the x axis showed no significant differences between groups, which could be explained by the fact that the task involves few movements on this axis.

Conclusion:

Accelerometer-based motion analysis is a useful tool to evaluate laparoscopic skill development of surgeons and should be used in training programs. Validation of this device in an in vivo setting is a research goal of the investigators' team.

Advanced Engineering Technology for Measuring Performance

The demand for competency-based assessments in surgical training is growing. Use of advanced engineering technology for clinical skills assessment allows for objective measures of hands-on performance. Clinical performance can be assessed in several ways via quantification of an assessee's hand movements (motion tracking), direction of visual attention (eye tracking), levels of stress (physiologic marker measurements), and location and pressure of palpation (force measurements). Innovations in video recording technology and qualitative analysis tools allow for a combination of observer- and technology-based assessments. Overall the goal is to create better assessments of surgical performance with robust validity evidence.

Technical skills assessment toolbox: a review using the unitary framework of validity

OBJECTIVE

The purpose of this study was to create a technical skills assessment toolbox for 35 basic and advanced skills/procedures that comprise the American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) surgical skills curriculum and to provide a critical appraisal of the included tools, using contemporary framework of validity.

BACKGROUND

Competency-based training has become the predominant model in surgical education and assessment of performance is an essential component. Assessment methods must produce valid results to accurately determine the level of competency.

METHODS

A search was performed, using PubMed and Google Scholar, to identify tools that have been developed for assessment of the targeted technical skills.

RESULTS

A total of 23 assessment tools for the 35 ACS/APDS skills modules were identified. Some tools, such as Operative Performance Rating System (OSATS) and Objective Structured Assessment of Technical Skill (OPRS), have been tested for more than 1 procedure. Therefore, 30 modules had at least 1 assessment tool, with some common surgical procedures being addressed by several tools. Five modules had none. Only 3 studies used Messick's framework to design their validity studies. The remaining studies used an outdated framework on the basis of "types of validity." When analyzed using the contemporary framework, few of these studies demonstrated validity for content, internal structure, and relationship to other variables.

CONCLUSIONS

This study provides an assessment toolbox for common surgical skills/procedures. Our review shows that few authors have used the contemporary unitary concept of validity for development of their assessment tools. As we progress toward competency-based training, future studies should provide evidence for various sources of validity using the contemporary framework.

Systematic instruction of arthroscopic knot tying with the ArK Trainer: an objective evaluation tool

PURPOSE

A Proficiency Formula was introduced as an objective self-evaluation method for evaluating basic arthroscopic knot tying in a laboratory setting. The correlation between the Proficiency Formula and gold standard pass/fail dichotomy was demonstrated, as well as with other popular evaluation tools--task-specific checklist (TSC) and global rating scale (GRS).

METHOD

A step-by-step video tutorial was used to instruct 35 medical students on how to tie an arthroscopic Samsung Medical Center (SMC) knot secured by three half hitches. Participants were video recorded performing arthroscopic knot tying and assessed on their success tying an SMC knot, pass or fail, and through three outcome tools: the Proficiency Formula, GRS and the TSC. Independent samples t test was used to compare the GRS, TSC and Proficiency Formula scores, between those who were passed or failed by the evaluators. Correlation between the measurement scales was tested using Spearman's rho correlation coefficient.

RESULTS

Participants received a mean proficiency score of 195 (140-249). The mean Proficiency score for those that passed was 323 (95 % CI 272-374), for those that failed, 87 (95 % CI 26-148, p < 0.001). We found strong linear correlation between the Proficiency Formula and GRS and TSE (0.83 and 0.78, respectively).

CONCLUSION

The Proficiency Formula has high correlation with gold standard GRS and TSC measurements when used to assess arthroscopic knot tying skills on a model. It has the added advantage of being able to be self-assessed.

Idle time: an underdeveloped performance metric for assessing surgical skill

BACKGROUND

The aim of this study was to evaluate validity evidence using idle time as a performance measure in open surgical skills assessment.

METHODS

This pilot study tested psychomotor planning skills of surgical attendings (n = 6), residents (n = 4) and medical students (n = 5) during suturing tasks of varying difficulty. Performance data were collected with a motion tracking system. Participants' hand movements were analyzed for idle time, total operative time, and path length. We hypothesized that there will be shorter idle times for more experienced individuals and on the easier tasks.

RESULTS

A total of 365 idle periods were identified across all participants. Attendings had fewer idle periods during 3 specific procedure steps (P < .001). All participants had longer idle time on friable tissue (P < .005).

CONCLUSIONS

Using an experimental model, idle time was found to correlate with experience and motor planning when operating on increasingly difficult tissue types. Further work exploring idle time as a valid psychomotor measure is warranted.

Assessment of lumbar puncture skill in experts and nonexperts using checklists and quantitative tracking of needle trajectories: implications for competency-based medical education

CONSTRUCT: With the current shift toward competency-based education, rigorous assessment tools are needed for procedurally based tasks.

BACKGROUND

Multiple tools exist to evaluate procedural skills, each with specific weaknesses.

APPROACH

We sought to determine if quantitative needle tracking could be used as a measure of lumbar puncture (LP) performance and added discriminatory value to a dichotomous checklist. Thirty-two medical students were divided into 2 groups. One group was asked to practice an LP once (single practice [SP]) and the other 5 times (multiple practice [MP]). Experts (attending ER physicians, senior ER residents, and a junior anesthesia resident) were used as comparators. Medical students were assessed again at 1 month to assess skill retention. Groups were assessed performing an LP with an electromagnetic tracking device that allows the needle's 3-dimensional movements to be captured and analyzed, and a dichotomous checklist.

RESULTS

Quantitative needle metrics as assessed by electromagnetic tracking showed a decreasing trend in needle movement distance with practice and with experience. The SP group made significantly more checklist mistakes initially as compared to the MP group (1.2 vs. 0.3, p <.05). At 1 month, there was a significant increase in both groups' mistakes (SP 3.4 vs. MP 1.3, p =.01). No correlation existed between individuals' needle motion and checklist mistakes.

CONCLUSIONS

These findings suggest that quantitative needle tracking identifies students who struggle with needle insertion but are successful at completing the dichotomous checklist.

Echocardiography practice: insights into appropriate clinical use, technical competence and quality improvement program

Echocardiography accounts for nearly half of all cardiac imaging techniques. It is a widely available and adaptable tool, as well as being a cost-effective and mainly a non-invasive test. In addition, echocardiography provides extensive clinical data, which is related to the presence or advent of different modalities (tissue Doppler imaging, speckle tracking imaging, three-dimensional mode, contrast echo, etc.), different approaches (transesophageal, intravascular, etc.), and different applications (ie, heart failure/resynchronization studies, ischemia/stress echo, etc.). In view of this, it is essential to conform to criteria of appropriate use and to keep standards of competence. In this study, we sought to review and discuss clinical practice of echocardiography in light of the criteria of appropriate clinical use, also we present an insight into echocardiographic technical competence and quality improvement project.

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting

BACKGROUND

Competence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.

METHODS

The study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.

RESULTS

Validity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.

CONCLUSIONS

We designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

EVA: laparoscopic instrument tracking based on Endoscopic Video Analysis for psychomotor skills assessment

INTRODUCTION

The EVA (Endoscopic Video Analysis) tracking system is a new system for extracting motions of laparoscopic instruments based on nonobtrusive video tracking. The feasibility of using EVA in laparoscopic settings has been tested in a box trainer setup.

METHODS

EVA makes use of an algorithm that employs information of the laparoscopic instrument's shaft edges in the image, the instrument's insertion point, and the camera's optical center to track the three-dimensional position of the instrument tip. A validation study of EVA comprised a comparison of the measurements achieved with EVA and the TrEndo tracking system. To this end, 42 participants (16 novices, 22 residents, and 4 experts) were asked to perform a peg transfer task in a box trainer. Ten motion-based metrics were used to assess their performance.

RESULTS

Construct validation of the EVA has been obtained for seven motion-based metrics. Concurrent validation revealed that there is a strong correlation between the results obtained by EVA and the TrEndo for metrics, such as path length (ρ = 0.97), average speed (ρ = 0.94), or economy of volume (ρ = 0.85), proving the viability of EVA.

CONCLUSIONS

EVA has been successfully validated in a box trainer setup, showing the potential of endoscopic video analysis to assess laparoscopic psychomotor skills. The results encourage further implementation of video tracking in training setups and image-guided surgery.

Relevance of motion-related assessment metrics in laparoscopic surgery

INTRODUCTION

Motion metrics have become an important source of information when addressing the assessment of surgical expertise. However, their direct relationship with the different surgical skills has not been fully explored. The purpose of this study is to investigate the relevance of motion-related metrics in the evaluation processes of basic psychomotor laparoscopic skills and their correlation with the different abilities sought to measure.

METHODS

A framework for task definition and metric analysis is proposed. An explorative survey was first conducted with a board of experts to identify metrics to assess basic psychomotor skills. Based on the output of that survey, 3 novel tasks for surgical assessment were designed. Face and construct validation was performed, with focus on motion-related metrics. Tasks were performed by 42 participants (16 novices, 22 residents, and 4 experts). Movements of the laparoscopic instruments were registered with the TrEndo tracking system and analyzed.

RESULTS

Time, path length, and depth showed construct validity for all 3 tasks. Motion smoothness and idle time also showed validity for tasks involving bimanual coordination and tasks requiring a more tactical approach, respectively. Additionally, motion smoothness and average speed showed a high internal consistency, proving them to be the most task-independent of all the metrics analyzed.

CONCLUSION

Motion metrics are complementary and valid for assessing basic psychomotor skills, and their relevance depends on the skill being evaluated. A larger clinical implementation, combined with quality performance information, will give more insight on the relevance of the results shown in this study.

Residents' perspectives of the value of a simulation curriculum in a general surgery residency program: a multimethod study of stakeholder feedback

BACKGROUND

Simulation has altered surgical curricula throughout residency programs. The purpose of this multimethod study was to explore residents' perceptions of simulation within surgical residency as relevant stakeholder feedback and program evaluation of the surgery simulation curriculum.

METHODS

Focus groups were held with a sample of surgery residents (n = 25) at a university-affiliated program. Residents participated in focus groups based on level of training and completed questionnaires regarding simulation curricula. Groups were facilitated by nonsurgeon faculty. Residents were asked: "What is the role of simulation in surgical education?" An interdisciplinary team recorded narrative data and performed content analyses. Quantitative data from questionnaires were summarized using descriptive statistics and frequencies.

RESULTS

Major themes from the qualitative data included: concerns regarding simulation in surgical education (28%), exposure to situations and technical skills in a low-stress learning environment (24%), pressure by external agencies (19%), an educational tool (17%), and quality assurance for patient care (12%). Laparoscopy and cadaver lab were the most prevalent simulation training during residency, in addition to trauma simulations, central lines/chest tubes/IV access, and stapling lab. In response to the statement: "ACGME should require a simulation curriculum in surgery residency," 52.1% responded favorably and 47.8% responded nonfavorably.

CONCLUSION

Residents acknowledge the value of simulation in patient safety, quality, and exposure to procedures before clinical experience, but remain divided on efficacy and requirement of simulation within curricula. The greater challenge to residency programs may be strategic implementation of simulation curricula within the right training context.

Ongoing professional performance evaluation (OPPE) using automatically captured electronic anesthesia data

BACKGROUND

The Massachusetts General Hospital (Boston), a large academic center providing anesthesia services for more than 49,000 procedures each year, created an Ongoing Professional Practice Evaluation (OPPE) process that could use readily available, automatically captured electronic information from its vendor-provided anesthesia information management system.

METHODS

The OPPE credentialing committee selected the following initial metrics: Blood pressure (BP) monitoring, end tidal CO2 monitoring, and timely documentation of compliance statements. Baseline data on the metrics were collected in an eight-month period (January 1, 2008-August 31, 2008). In February 2009 information on the metrics was provided to the department's staff members, and the ongoing evaluation process began. On the basis of three months of data, final reports for physicians being credentialed were distributed. Each report included a listing for each metric of the total number of compliant cases and noncompliant cases and a comparison by percentage to the baseline departmental evaluation. A summary statement indicated whether a physician's performance was within the group representing 95% of all department physicians. Noncompliant cases were listed by medical record number and case date so providers and reviewers could examine individual cases.

CONCLUSION

A novel, automated, and continuous reporting system for physician credentialing that uses the existing clinical information system infrastructure can serve as a key element of a comprehensive clinical performance evaluation that measures both technical and generalizable clinical skill sets. It is not intended to provide a complete system for measuring competence but rather to serve as a first-round warning mechanism and metric scoring tool to identify problems and potential performance noncompliance issues.

Randomized controlled trial of virtual reality and hybrid simulation for robotic surgical training

OBJECTIVE

To evaluate if two commonly used laparoscopic simulators could be adapted and used successfully for the robotics platform in a laparoscopic and robotic naïve medical student population.

MATERIALS AND METHODS

We identified two widely validated laparoscopic simulation programs, LapSim(®) (Surgical Science Sweden AB), and ProMIS(®) (Haptica, Ireland)for inclusion in the study. The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills(®) task set was used for ProMIS, and adapted for the DaVinci(®) console (Intuitive Surgical, Inc., Sunnyvale, CA, USA) robotic platform. We then randomized 20 naïve medical students to receive training on either LapSim or ProMIS, both or neither, and evaluated them before and after training.

RESULTS

When the groups were compared at baseline, there were no statistical differences in mean scores amongst the groups in univariate analysis (α= 0.05). When comparing mean scores within groups before and after training sessions, statistically significant performance enhancement in all four robotic tasks were identified in the groups receiving dual training.

CONCLUSION

We have shown that the use of ProMIS hybrid and LapSim virtual reality (VR) simulators in conjunction with each other can considerable improve robotic console performance in novice medical students compared with hybrid and VR simulation alone.