Development and content validation of performance assessments for endoscopic third ventriculostomy

Content and Face Validation of a Novel, Interactive Nutrition Specific Physical Exam Competency Tool (INSPECT) to Evaluate Registered Dietitians' Competence: A Delphi Consensus from the United States

The nutrition-focused physical examination (NFPE) is an integral component of nutrition assessment performed by registered dietitian nutritionists (RDNs) to determine signs of malnutrition and other nutrition-related complications. Increased use of this essential skill among RDNs and the transformation of dietetics education to a competency-based model in the near future calls for appropriately validated tools to measure RDNs' NFPE competence. To fill the need for a validated competency tool, this study developed an Interactive Nutrition-Specific Physical Exam Competency Tool (INSPECT) utilizing the initial 70 items identified in the first phase of the study. The second phase of this study aimed to test the preliminary version of the INSPECT for content and face validity. An expert panel of 17 members provided consensus recommendations through the Delphi process. Internal consistency of the consensus was measured with Cronbach's alpha (α) and α of ≥0.70 was defined as acceptable a priori. Inter-rater agreement among the expert panel was determined using the intraclass correlation coefficient (ICC) and an a priori ICC of 0.75 to 0.9 was established as good and >0.9 as excellent agreement. The results showed acceptable face validity (α = 0.71) and excellent content validity for the INSPECT, with an internal consistency of α = 0.97 in the first round and α = 0.96 in the second round. The inter-rater agreement was also excellent with ICC = 0.95 for each of the Delphi rounds. A total of 52 items were retained from the preliminary version of the INSPECT. Open feedback from the experts allowed for the consolidation of 11 similar items for better scoring and evaluation and thus, a total of 41 items were included in the final version of the INSPECT. The final version of the INSPECT is currently being studied in real-life, multi-site clinical settings among practicing RDNs to examine construct validity, reliability, and item-level psychometric properties. Ultimately, the validated INSPECT will be available for the competency evaluation of RDNs practicing in clinical settings.

Video-Rated Performance Assessment of Simulated Laparoscopic Sleeve Gastrectomy: Validation of a Sleeve Gastrectomy Rating Scale

PURPOSE

The global rise in obesity has been accompanied by widespread uptake of the procedure of laparoscopic sleeve gastrectomy. Despite this, the key components for performance assessment have not been standardized for this procedure. The aim of this study was to develop and demonstrate the validity of a Sleeve Objective Structured Assessment of Technical Skill (SOSATS) scale for learning the procedure of laparoscopic sleeve gastrectomy (LSG).

MATERIALS AND METHODS

The SOSATS evaluation tool was based upon critical steps of the LSG procedure. Both the SOSATS and the Global Rating Scale (GRS) component of the Objective Structured Assessment of Technical Skill (OSATS) tools were utilized in a prospective single-blinded observational study design of 26 video recordings of surgeons performing sleeve gastrectomies using a novel simulation. The surgeons were allocated into "novice" or "experienced" groups dependent on case-volume criteria. Surgical performance was assessed using both the GRS and SOSATS scales by blinded assessors of the video recordings.

RESULTS

Face and content validity were demonstrated for key components of the simulated model. An overall positive correlation was established inferring concurrent validity between the accepted OSATS Global Rating Scale against the SOSATS procedural scale. Construct validity was established for a number of areas of the SOSATS scale.

CONCLUSION

The SOSATS scale is shown to exhibit construct and concurrent validity in the simulated setting for the procedure of sleeve gastrectomy. Utilizing this scale to review surgical performance is potentially feasible and reliable but would require further research prior to use in high-stakes assessment processes such as credentialing.

A Review of Physical Simulators for Neuroendoscopy Skills Training

BACKGROUND

Minimally invasive neurosurgical approaches reduce patient morbidity by providing the surgeon with better visualization and access to complex lesions, with minimal disruption to normal anatomy. The use of rigid or flexible neuroendoscopes, supplemented with a conventional stereoscopic operating microscope, has been integral to the adoption of these techniques. Neurosurgeons commonly use neuroendoscopes to perform the ventricular and endonasal approaches. It is challenging to learn neuroendoscopy skills from the existing apprenticeship model of surgical education. The training methods, which use simulation-based systems, have achieved wide acceptance. Physical simulators provide anatomic orientation and hands-on experience with repeatability. Our aim is to review the existing physical simulators on the basis of the skills training of neuroendoscopic procedures.

METHODS

We searched Scopus, Google Scholar, PubMed, IEEE Xplore, and dblp. We used the following keywords "neuroendoscopy," "training," "simulators," "physical," and "skills evaluation." A total of 351 articles were screened based on development methods, evaluation criteria, and validation studies on physical simulators for skills training in neuroendoscopy.

RESULTS

The screening of the articles resulted in classifying the physical training methods developed for neuroendoscopy surgical skills into synthetic simulators and box trainers. The existing simulators were compared based on their design, fidelity, trainee evaluation methods, and validation studies.

CONCLUSIONS

The state of simulation systems demands collaborative initiatives among translational research institutes. They need improved fidelity and validation studies for inclusion in the surgical educational curriculum. Learning should be imparted in stages with standardization of performance metrics for skills evaluation.

New Simulator for Neuroendoscopy: A Realistic and Attainable Model

OBJECTIVE

To present an attainable and realistic model for neuroendoscopic simulation which replicates exercises of tissue biopsy and coagulation and membrane fenestration.

METHODS

We presented a stepwise method to create a neuroendoscopic simulation model using bovine brain and membrane units made by a soda cup covered by an amniotic membrane inside an expanded polystyrene spherical container. We used face validation for preliminary evaluation. We also rated the students before and after training with the NEVAT global rating scale (GRS) and recorded the time required to complete all 3 procedures (third ventriculostomy, tissue biopsy, and coagulation). The total cost of the model was $5.

RESULTS

The experts consider this new model as capable of reproducing real surgical situations with great similarity to the human brain. We tested the model in 20 trainees. The median GRS score before the training was 9 (range, 7-12). After repeated training and performance feedback, the final median GRS score was 41 (range, 37.5-45; P < 0.0001). The time needed to finish the exercises before training was 33 minutes (range, 30.5-42.5 minutes), and after using the model the final median time was 20 minutes (range, 17.5-22 minutes; P < 0.0001).

CONCLUSIONS

Simulators for neuroendoscopy described so far are reliable, but they entail a high cost. Models with live animals, although of lower cost, are questioned from an ethical point of view. In the current work, we describe a high fidelity ventricular neuroendoscopic simulator model that, because of its low cost, can be replicated in any training center that has a neuroendoscope.

Simulation-based Education for Endoscopic Third Ventriculostomy: A Comparison Between Virtual and Physical Training Models

BACKGROUND

The relative educational benefits of virtual reality (VR) and physical simulation models for endoscopic third ventriculostomy (ETV) have not been evaluated "head to head."

OBJECTIVE

To compare and identify the relative utility of a physical and VR ETV simulation model for use in neurosurgical training.

METHODS

Twenty-three neurosurgical residents and 3 fellows performed an ETV on both a physical and VR simulation model. Trainees rated the models using 5-point Likert scales evaluating the domains of anatomy, instrument handling, procedural content, and the overall fidelity of the simulation. Paired t tests were performed for each domain's mean overall score and individual items.

RESULTS

The VR model has relative benefits compared with the physical model with respect to realistic representation of intraventricular anatomy at the foramen of Monro (4.5, standard deviation [SD] = 0.7 vs 4.1, SD = 0.6; P = .04) and the third ventricle floor (4.4, SD = 0.6 vs 4.0, SD = 0.9; P = .03), although the overall anatomy score was similar (4.2, SD = 0.6 vs 4.0, SD = 0.6; P = .11). For overall instrument handling and procedural content, the physical simulator outperformed the VR model (3.7, SD = 0.8 vs 4.5; SD = 0.5, P < .001 and 3.9; SD = 0.8 vs 4.2, SD = 0.6; P = .02, respectively). Overall task fidelity across the 2 simulators was not perceived as significantly different.

CONCLUSION

Simulation model selection should be based on educational objectives. Training focused on learning anatomy or decision-making for anatomic cues may be aided with the VR simulation model. A focus on developing manual dexterity and technical skills using endoscopic equipment in the operating room may be better learned on the physical simulation model.

Validity Evidence for the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT)

BACKGROUND

Growing demand for transparent and standardized methods for evaluating surgical competence prompted the construction of the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).

OBJECTIVE

To provide validity evidence of the NEVAT by reporting on the tool's internal structure and its relationship with surgical expertise during simulation-based training.

METHODS

The NEVAT was used to assess performance of trainees and faculty at an international neuroendoscopy workshop. All participants performed an endoscopic third ventriculostomy (ETV) on a synthetic simulator. Participants were simultaneously scored by 2 raters using the NEVAT procedural checklist and global rating scale (GRS). Evidence of internal structure was collected by calculating interrater reliability and internal consistency of raters' scores. Evidence of relationships with other variables was collected by comparing the ETV performance of experts, experienced trainees, and novices using Jonckheere's test (evidence of construct validity).

RESULTS

Thirteen experts, 11 experienced trainees, and 10 novices participated. The interrater reliability by the intraclass correlation coefficient for the checklist and GRS was 0.82 and 0.94, respectively. Internal consistency (Cronbach's α) for the checklist and the GRS was 0.74 and 0.97, respectively. Median scores with interquartile range on the checklist and GRS for novices, experienced trainees, and experts were 0.69 (0.58-0.86), 0.85 (0.63-0.89), and 0.85 (0.81-0.91) and 3.1 (2.5-3.8), 3.7 (2.2-4.3) and 4.6 (4.4-4.9), respectively. Jonckheere's test showed that the median checklist and GRS score increased with performer expertise ( P = .04 and .002, respectively).

CONCLUSION

This study provides validity evidence for the NEVAT to support its use as a standardized method of evaluating neuroendoscopic competence during simulation-based training.

Development of synthetic simulators for endoscope-assisted repair of metopic and sagittal craniosynostosis

OBJECTIVE Endoscope-assisted repair of craniosynostosis is a safe and efficacious alternative to open techniques. However, this procedure is challenging to learn, and there is significant variation in both its execution and outcomes. Surgical simulators may allow trainees to learn and practice this procedure prior to operating on an actual patient. The purpose of this study was to develop a realistic, relatively inexpensive simulator for endoscope-assisted repair of metopic and sagittal craniosynostosis and to evaluate the models' fidelity and teaching content. METHODS Two separate, 3D-printed, plastic powder-based replica skulls exhibiting metopic (age 1 month) and sagittal (age 2 months) craniosynostosis were developed. These models were made into consumable skull "cartridges" that insert into a reusable base resembling an infant's head. Each cartridge consists of a multilayer scalp (skin, subcutaneous fat, galea, and periosteum); cranial bones with accurate landmarks; and the dura mater. Data related to model construction, use, and cost were collected. Eleven novice surgeons (residents), 9 experienced surgeons (fellows), and 5 expert surgeons (attendings) performed a simulated metopic and sagittal craniosynostosis repair using a neuroendoscope, high-speed drill, rongeurs, lighted retractors, and suction/irrigation. All participants completed a 13-item questionnaire (using 5-point Likert scales) to rate the realism and utility of the models for teaching endoscope-assisted strip suturectomy. RESULTS The simulators are compact, robust, and relatively inexpensive. They can be rapidly reset for repeated use and contain a minimal amount of consumable material while providing a realistic simulation experience. More than 80% of participants agreed or strongly agreed that the models' anatomical features, including surface anatomy, subgaleal and subperiosteal tissue planes, anterior fontanelle, and epidural spaces, were realistic and contained appropriate detail. More than 90% of participants indicated that handling the endoscope and the instruments was realistic, and also that the steps required to perform the procedure were representative of the steps required in real life. CONCLUSIONS Both the metopic and sagittal craniosynostosis simulators were developed using low-cost methods and were successfully designed to be reusable. The simulators were found to realistically represent the surgical procedure and can be used to develop the technical skills required for performing an endoscope-assisted craniosynostosis repair.

Three-Dimensional Imaging Anatomic Study and Clinical Application of the Third Ventricle Transcallosal-Transforniceal Approach

The third ventricle is a narrow, funnel-shaped, unilocular, midline cavity located between the 2 thalami, under the body of the lateral ventricle, in the center of the head. Damaging of brain tissue in the third ventricle when conducting operation under the microscope will lead to serious consequence. The study aimed to precisely detect the relative location of specific structures on the approach to the third ventricle. The authors rebuilt a 3-dimensional reconstruction of the brain and selected specific sections and then measured several crucial distance, angle to precisely assure the approaching pathway and localize the hypothalamic sulcus, interthalamic adhesion, anterior commissure, optic chiasm, and pineal body. In the study, canthomeatal line was used as base line to measure angle. Parameters were obtained from 58 samples (22 males and 36 females) between 21 and 76 years old. Means and standard deviation were calculated as well as the 95% confidence interval for the mean value of the measured data. The data were analyzed by SPSS, statistical software with the comparison between sexes and sides. The results could be reference for clinical and anatomic utilities.