Stand in surgeon's shoes: virtual reality cross-training to enhance teamwork in surgery

PURPOSE

Teamwork in surgery depends on a shared mental model of success, i.e., a common understanding of objectives in the operating room. A shared model leads to increased engagement among team members and is associated with fewer complications and overall better outcomes for patients. However, clinical training typically focuses on role-specific skills, leaving individuals to acquire a shared model indirectly through on-the-job experience.

METHODS

We investigate whether virtual reality (VR) cross-training, i.elet@tokeneonedotexposure to other roles, can enhance a shared mental model for non-surgeons more directly. Our study focuses on X-ray guided pelvic trauma surgery, a procedure where successful communication depends on the shared model between the surgeon and a C-arm technologist. We present a VR environment supporting both roles and evaluate a cross-training curriculum in which non-surgeons swap roles with the surgeon.

RESULTS

Exposure to the surgical task resulted in higher engagement with the C-arm technologist role in VR, as measured by the mental demand and effort expended by participants ( p < 0.001 ). It also has a significant effect on non-surgeon's mental model of the overall task; novice participants' estimation of the mental demand and effort required for the surgeon's task increases after training, while their perception of overall performance decreases ( p < 0.05 ), indicating a gap in understanding based solely on observation. This phenomenon was also present for a professional C-arm technologist.

CONCLUSION

Until now, VR applications for clinical training have focused on virtualizing existing curricula. We demonstrate how novel approaches which are not possible outside of a virtual environment, such as role swapping, may enhance the shared mental model of surgical teams by contextualizing each individual's role within the overall task in a time- and cost-efficient manner. As workflows grow increasingly sophisticated, we see VR curricula as being able to directly foster a shared model for success, ultimately benefiting patient outcomes through more effective teamwork in surgery.

Tremor Assessment in Robot-Assisted Microlaryngeal Surgery Using Computer Vision-Based Tool Tracking

OBJECTIVE

Use microscopic video-based tracking of laryngeal surgical instruments to investigate the effect of robot assistance on instrument tremor.

STUDY DESIGN

Experimental trial.

SETTING

Tertiary Academic Medical Center.

METHODS

In this randomized cross-over trial, 36 videos were recorded from 6 surgeons performing left and right cordectomies on cadaveric pig larynges. These recordings captured 3 distinct conditions: without robotic assistance, with robot-assisted scissors, and with robot-assisted graspers. To assess tool tremor, we employed computer vision-based algorithms for tracking surgical tools. Absolute tremor bandpower and normalized path length were utilized as quantitative measures. Wilcoxon rank sum exact tests were employed for statistical analyses and comparisons between trials. Additionally, surveys were administered to assess the perceived ease of use of the robotic system.

RESULTS

Absolute tremor bandpower showed a significant decrease when using robot-assisted instruments compared to freehand instruments (P = .012). Normalized path length significantly decreased with robot-assisted compared to freehand trials (P = .001). For the scissors, robot-assisted trials resulted in a significant decrease in absolute tremor bandpower (P = .002) and normalized path length (P < .001). For the graspers, there was no significant difference in absolute tremor bandpower (P = .4), but there was a significantly lower normalized path length in the robot-assisted trials (P = .03).

CONCLUSION

This study demonstrated that computer-vision-based approaches can be used to assess tool motion in simulated microlaryngeal procedures. The results suggest that robot assistance is capable of reducing instrument tremor.

Arc-to-line frame registration method for ultrasound and photoacoustic image-guided intraoperative robot-assisted laparoscopic prostatectomy

PURPOSE

To achieve effective robot-assisted laparoscopic prostatectomy, the integration of transrectal ultrasound (TRUS) imaging system which is the most widely used imaging modality in prostate imaging is essential. However, manual manipulation of the ultrasound transducer during the procedure will significantly interfere with the surgery. Therefore, we propose an image co-registration algorithm based on a photoacoustic marker (PM) method, where the ultrasound/photoacoustic (US/PA) images can be registered to the endoscopic camera images to ultimately enable the TRUS transducer to automatically track the surgical instrument.

METHODS

An optimization-based algorithm is proposed to co-register the images from the two different imaging modalities. The principle of light propagation and an uncertainty in PM detection were assumed in this algorithm to improve the stability and accuracy of the algorithm. The algorithm is validated using the previously developed US/PA image-guided system with a da Vinci surgical robot.

RESULTS

The target-registration-error (TRE) is measured to evaluate the proposed algorithm. In both simulation and experimental demonstration, the proposed algorithm achieved a sub-centimeter accuracy which is acceptable in practical clinics (i.e., 1.15 ± 0.29 mm from the experimental evaluation). The result is also comparable with our previous approach (i.e., 1.05 ± 0.37 mm), and the proposed method can be implemented with a normal white light stereo camera and does not require highly accurate localization of the PM.

CONCLUSION

The proposed frame registration algorithm enabled a simple yet efficient integration of commercial US/PA imaging system into laparoscopic surgical setting by leveraging the characteristic properties of acoustic wave propagation and laser excitation, contributing to automated US/PA image-guided surgical intervention applications.

Interactive Multi-Stage Robotic Positioner for Intra-Operative MRI-Guided Stereotactic Neurosurgery

Magnetic resonance imaging (MRI) demonstrates clear advantages over other imaging modalities in neurosurgery with its ability to delineate critical neurovascular structures and cancerous tissue in high-resolution 3D anatomical roadmaps. However, its application has been limited to interventions performed based on static pre/post-operative imaging, where errors accrue from stereotactic frame setup, image registration, and brain shift. To leverage the powerful intra-operative functions of MRI, e.g., instrument tracking, monitoring of physiological changes and tissue temperature in MRI-guided bilateral stereotactic neurosurgery, a multi-stage robotic positioner is proposed. The system positions cannula/needle instruments using a lightweight (203 g) and compact (Ø97 × 81 mm) skull-mounted structure that fits within most standard imaging head coils. With optimized design in soft robotics, the system operates in two stages: i) manual coarse adjustment performed interactively by the surgeon (workspace of ±30°), ii) automatic fine adjustment with precise (<0.2° orientation error), responsive (1.4 Hz bandwidth), and high-resolution (0.058°) soft robotic positioning. Orientation locking provides sufficient transmission stiffness (4.07 N/mm) for instrument advancement. The system's clinical workflow and accuracy is validated with lab-based (<0.8 mm) and MRI-based testing on skull phantoms (<1.7 mm) and a cadaver subject (<2.2 mm). Custom-made wireless omni-directional tracking markers facilitated robot registration under MRI.

Steady-Hand Eye Robot 3.0: Optimization and Benchtop Evaluation for Subretinal Injection

Subretinal injection methods and other procedures for treating retinal conditions and diseases (many considered incurable) have been limited in scope due to limited human motor control. This study demonstrates the next generation, cooperatively controlled Steady-Hand Eye Robot (SHER 3.0), a precise and intuitive-to-use robotic platform achieving clinical standards for targeting accuracy and resolution for subretinal injections. The system design and basic kinematics are reported and a deflection model for the incorporated delta stage and validation experiments are presented. This model optimizes the delta stage parameters, maximizing the global conditioning index and minimizing torsional compliance. Five tests measuring accuracy, repeatability, and deflection show the optimized stage design achieves a tip accuracy of < 30 μm, tip repeatability of 9.3 μm and 0.02°, and deflections between 20-350 μm/N. Future work will use updated control models to refine tip positioning outcomes and will be tested on in vivo animal models.

Autonomous Spinal Robotic System for Transforaminal Lumbar Epidural Injections: A Proof of Concept of Study

STUDY DESIGN

Phantom study.

OBJECTIVE

The aim of our study is to demonstrate in a proof-of-concept model whether the use of a marker less autonomous robotic controlled injection delivery system will increase accuracy in the lumbar spine.

METHODS

Ideal transforaminal epidural injection trajectories (bilateral L2/3, L3/4, L4/5, L5/S1 and S1) were planned out on a virtual pre-operative planning software by 1 experienced provider. Twenty transforaminal epidural injections were administered in a lumbar spine phantom model, 10 using a freehand procedure, and 10 using a marker less autonomous spinal robotic system. Procedural accuracy, defined as the difference between pre-operative planning and actual post-operative needle tip distance (mm) and angular orientation (degrees), were assessed between the freehand and robotic procedures.

RESULTS

Procedural accuracy for robotically placed transforaminal epidural injections was significantly higher with the difference in pre- and post-operative needle tip distance being 20.1 (±5.0) mm in the freehand procedure and 11.4 (±3.9) mm in the robotically placed procedure (P < .001). Needle tip precision for the freehand technique was 15.6 mm (26.3 - 10.7) compared to 10.1 mm (16.3 - 6.1) for the robotic technique. Differences in needle angular orientation deviation were 5.6 (±3.3) degrees in the robotically placed procedure and 12.0 (±4.8) degrees in the freehand procedure (P = .003).

CONCLUSION

The robotic system allowed for comparable placement of transforaminal epidural injections as a freehand technique by an experienced provider, with additional benefits of improved accuracy and precision.

Fully immersive virtual reality for skull-base surgery: surgical training and beyond

PURPOSE

A virtual reality (VR) system, where surgeons can practice procedures on virtual anatomies, is a scalable and cost-effective alternative to cadaveric training. The fully digitized virtual surgeries can also be used to assess the surgeon's skills using measurements that are otherwise hard to collect in reality. Thus, we present the Fully Immersive Virtual Reality System (FIVRS) for skull-base surgery, which combines surgical simulation software with a high-fidelity hardware setup.

METHODS

FIVRS allows surgeons to follow normal clinical workflows inside the VR environment. FIVRS uses advanced rendering designs and drilling algorithms for realistic bone ablation. A head-mounted display with ergonomics similar to that of surgical microscopes is used to improve immersiveness. Extensive multi-modal data are recorded for post-analysis, including eye gaze, motion, force, and video of the surgery. A user-friendly interface is also designed to ease the learning curve of using FIVRS.

RESULTS

We present results from a user study involving surgeons with various levels of expertise. The preliminary data recorded by FIVRS differentiate between participants with different levels of expertise, promising future research on automatic skill assessment. Furthermore, informal feedback from the study participants about the system's intuitiveness and immersiveness was positive.

CONCLUSION

We present FIVRS, a fully immersive VR system for skull-base surgery. FIVRS features a realistic software simulation coupled with modern hardware for improved realism. The system is completely open source and provides feature-rich data in an industry-standard format.

A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers

Image-based 2D/3D registration is a critical technique for fluoroscopic guided surgical interventions. Conventional intensity-based 2D/3D registration approa- ches suffer from a limited capture range due to the presence of local minima in hand-crafted image similarity functions. In this work, we aim to extend the 2D/3D registration capture range with a fully differentiable deep network framework that learns to approximate a convex-shape similarity function. The network uses a novel Projective Spatial Transformer (ProST) module that has unique differentiability with respect to 3D pose parameters, and is trained using an innovative double backward gradient-driven loss function. We compare the most popular learning-based pose regression methods in the literature and use the well-established CMAES intensity-based registration as a benchmark. We report registration pose error, target registration error (TRE) and success rate (SR) with a threshold of 10mm for mean TRE. For the pelvis anatomy, the median TRE of ProST followed by CMAES is 4.4mm with a SR of 65.6% in simulation, and 2.2mm with a SR of 73.2% in real data. The CMAES SRs without using ProST registration are 28.5% and 36.0% in simulation and real data, respectively. Our results suggest that the proposed ProST network learns a practical similarity function, which vastly extends the capture range of conventional intensity-based 2D/3D registration. We believe that the unique differentiable property of ProST has the potential to benefit related 3D medical imaging research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.

A Self-Configuring Deep Learning Network for Segmentation of Temporal Bone Anatomy in Cone-Beam CT Imaging

OBJECTIVE

Preoperative planning for otologic or neurotologic procedures often requires manual segmentation of relevant structures, which can be tedious and time-consuming. Automated methods for segmenting multiple geometrically complex structures can not only streamline preoperative planning but also augment minimally invasive and/or robot-assisted procedures in this space. This study evaluates a state-of-the-art deep learning pipeline for semantic segmentation of temporal bone anatomy.

STUDY DESIGN

A descriptive study of a segmentation network.

SETTING

Academic institution.

METHODS

A total of 15 high-resolution cone-beam temporal bone computed tomography (CT) data sets were included in this study. All images were co-registered, with relevant anatomical structures (eg, ossicles, inner ear, facial nerve, chorda tympani, bony labyrinth) manually segmented. Predicted segmentations from no new U-Net (nnU-Net), an open-source 3-dimensional semantic segmentation neural network, were compared against ground-truth segmentations using modified Hausdorff distances (mHD) and Dice scores.

RESULTS

Fivefold cross-validation with nnU-Net between predicted and ground-truth labels were as follows: malleus (mHD: 0.044 ± 0.024 mm, dice: 0.914 ± 0.035), incus (mHD: 0.051 ± 0.027 mm, dice: 0.916 ± 0.034), stapes (mHD: 0.147 ± 0.113 mm, dice: 0.560 ± 0.106), bony labyrinth (mHD: 0.038 ± 0.031 mm, dice: 0.952 ± 0.017), and facial nerve (mHD: 0.139 ± 0.072 mm, dice: 0.862 ± 0.039). Comparison against atlas-based segmentation propagation showed significantly higher Dice scores for all structures (p < .05).

CONCLUSION

Using an open-source deep learning pipeline, we demonstrate consistently submillimeter accuracy for semantic CT segmentation of temporal bone anatomy compared to hand-segmented labels. This pipeline has the potential to greatly improve preoperative planning workflows for a variety of otologic and neurotologic procedures and augment existing image guidance and robot-assisted systems for the temporal bone.

Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation

Surgical phase recognition (SPR) is a crucial element in the digital transformation of the modern operating theater. While SPR based on video sources is well-established, incorporation of interventional X-ray sequences has not yet been explored. This paper presents Pelphix, a first approach to SPR for X-ray-guided percutaneous pelvic fracture fixation, which models the procedure at four levels of granularity - corridor, activity, view, and frame value - simulating the pelvic fracture fixation workflow as a Markov process to provide fully annotated training data. Using added supervision from detection of bony corridors, tools, and anatomy, we learn image representations that are fed into a transformer model to regress surgical phases at the four granularity levels. Our approach demonstrates the feasibility of X-ray-based SPR, achieving an average accuracy of 99.2% on simulated sequences and 71.7% in cadaver across all granularity levels, with up to 84% accuracy for the target corridor in real data. This work constitutes the first step toward SPR for the X-ray domain, establishing an approach to categorizing phases in X-ray-guided surgery, simulating realistic image sequences to enable machine learning model development, and demonstrating that this approach is feasible for the analysis of real procedures. As X-ray-based SPR continues to mature, it will benefit procedures in orthopedic surgery, angiography, and interventional radiology by equipping intelligent surgical systems with situational awareness in the operating room.

An autonomous X-ray image acquisition and interpretation system for assisting percutaneous pelvic fracture fixation

PURPOSE

Percutaneous fracture fixation involves multiple X-ray acquisitions to determine adequate tool trajectories in bony anatomy. In order to reduce time spent adjusting the X-ray imager's gantry, avoid excess acquisitions, and anticipate inadequate trajectories before penetrating bone, we propose an autonomous system for intra-operative feedback that combines robotic X-ray imaging and machine learning for automated image acquisition and interpretation, respectively.

METHODS

Our approach reconstructs an appropriate trajectory in a two-image sequence, where the optimal second viewpoint is determined based on analysis of the first image. A deep neural network is responsible for detecting the tool and corridor, here a K-wire and the superior pubic ramus, respectively, in these radiographs. The reconstructed corridor and K-wire pose are compared to determine likelihood of cortical breach, and both are visualized for the clinician in a mixed reality environment that is spatially registered to the patient and delivered by an optical see-through head-mounted display.

RESULTS

We assess the upper bounds on system performance through in silico evaluation across 11 CTs with fractures present, in which the corridor and K-wire are adequately reconstructed. In post hoc analysis of radiographs across 3 cadaveric specimens, our system determines the appropriate trajectory to within 2.8 ± 1.3 mm and 2.7 ± 1.8[Formula: see text].

CONCLUSION

An expert user study with an anthropomorphic phantom demonstrates how our autonomous, integrated system requires fewer images and lower movement to guide and confirm adequate placement compared to current clinical practice. Code and data are available.

TAToo: vision-based joint tracking of anatomy and tool for skull-base surgery

PURPOSE

Tracking the 3D motion of the surgical tool and the patient anatomy is a fundamental requirement for computer-assisted skull-base surgery. The estimated motion can be used both for intra-operative guidance and for downstream skill analysis. Recovering such motion solely from surgical videos is desirable, as it is compliant with current clinical workflows and instrumentation.

METHODS

We present Tracker of Anatomy and Tool (TAToo). TAToo jointly tracks the rigid 3D motion of the patient skull and surgical drill from stereo microscopic videos. TAToo estimates motion via an iterative optimization process in an end-to-end differentiable form. For robust tracking performance, TAToo adopts a probabilistic formulation and enforces geometric constraints on the object level.

RESULTS

We validate TAToo on both simulation data, where ground truth motion is available, as well as on anthropomorphic phantom data, where optical tracking provides a strong baseline. We report sub-millimeter and millimeter inter-frame tracking accuracy for skull and drill, respectively, with rotation errors below [Formula: see text]. We further illustrate how TAToo may be used in a surgical navigation setting.

CONCLUSIONS

We present TAToo, which simultaneously tracks the surgical tool and the patient anatomy in skull-base surgery. TAToo directly predicts the motion from surgical videos, without the need of any markers. Our results show that the performance of TAToo compares favorably to competing approaches. Future work will include fine-tuning of our depth network to reach a 1 mm clinical accuracy goal desired for surgical applications in the skull base.

A force-sensing surgical drill for real-time force feedback in robotic mastoidectomy

PURPOSE

Robotic assistance in otologic surgery can reduce the task load of operating surgeons during the removal of bone around the critical structures in the lateral skull base. However, safe deployment into the anatomical passageways necessitates the development of advanced sensing capabilities to actively limit the interaction forces between the surgical tools and critical anatomy.

METHODS

We introduce a surgical drill equipped with a force sensor that is capable of measuring accurate tool-tissue interaction forces to enable force control and feedback to surgeons. The design, calibration and validation of the force-sensing surgical drill mounted on a cooperatively controlled surgical robot are described in this work.

RESULTS

The force measurements on the tip of the surgical drill are validated with raw-egg drilling experiments, where a force sensor mounted below the egg serves as ground truth. The average root mean square error for points and path drilling experiments is 41.7 (± 12.2) mN and 48.3 (± 13.7) mN, respectively.

CONCLUSION

The force-sensing prototype measures forces with sub-millinewton resolution and the results demonstrate that the calibrated force-sensing drill generates accurate force measurements with minimal error compared to the measured drill forces. The development of such sensing capabilities is crucial for the safe use of robotic systems in a clinical context.

Platform for investigating continuum manipulator behavior in orthopedics

PURPOSE

The use of robotic continuum manipulators has been proposed to facilitate less-invasive orthopedic surgical procedures. While tools and strategies have been developed, critical challenges such as system control and intra-operative guidance are under-addressed. Simulation tools can help solve these challenges, but several gaps limit their utility for orthopedic surgical systems, particularly those with continuum manipulators. Herein, a simulation platform which addresses these gaps is presented as a tool to better understand and solve challenges for minimally invasive orthopedic procedures.

METHODS

An open-source surgical simulation software package was developed in which a continuum manipulator can interact with any volume model such as to drill bone volumes segmented from a 3D computed tomography (CT) image. Paired simulated X-ray images of the scene can also be generated. As compared to previous works, tool-anatomy interactions use a physics-based approach which leads to more stable behavior and wider procedure applicability. A new method for representing low-level volumetric drilling behavior is also introduced to capture material variability within bone as well as patient-specific properties from a CT.

RESULTS

Similar interaction between a continuum manipulator and phantom bone was also demonstrated between a simulated manipulator and volumetric obstacle models. High-level material- and tool-driven behavior was shown to emerge directly from the improved low-level interactions, rather than by need of manual programming.

CONCLUSION

This platform is a promising tool for developing and investigating control algorithms for tasks such as curved drilling. The generation of simulated X-ray images that correspond to the scene is useful for developing and validating image guidance models. The improvements to volumetric drilling offer users the ability to better tune behavior for specific tools and procedures and enable research to improve surgical simulation model fidelity. This platform will be used to develop and test control algorithms for image-guided curved drilling procedures in the femur.

Investigating keypoint descriptors for camera relocalization in endoscopy surgery

PURPOSE

Recent advances in computer vision and machine learning have resulted in endoscopic video-based solutions for dense reconstruction of the anatomy. To effectively use these systems in surgical navigation, a reliable image-based technique is required to constantly track the endoscopic camera's position within the anatomy, despite frequent removal and re-insertion. In this work, we investigate the use of recent learning-based keypoint descriptors for six degree-of-freedom camera pose estimation in intraoperative endoscopic sequences and under changes in anatomy due to surgical resection.

METHODS

Our method employs a dense structure from motion (SfM) reconstruction of the preoperative anatomy, obtained with a state-of-the-art patient-specific learning-based descriptor. During the reconstruction step, each estimated 3D point is associated with a descriptor. This information is employed in the intraoperative sequences to establish 2D-3D correspondences for Perspective-n-Point (PnP) camera pose estimation. We evaluate this method in six intraoperative sequences that include anatomical modifications obtained from two cadaveric subjects.

RESULTS

Show that this approach led to translation and rotation errors of 3.9 mm and 0.2 radians, respectively, with 21.86% of localized cameras averaged over the six sequences. In comparison to an additional learning-based descriptor (HardNet++), the selected descriptor can achieve a better percentage of localized cameras with similar pose estimation performance. We further discussed potential error causes and limitations of the proposed approach.

CONCLUSION

Patient-specific learning-based descriptors can relocalize images that are well distributed across the inspected anatomy, even where the anatomy is modified. However, camera relocalization in endoscopic sequences remains a persistently challenging problem, and future research is necessary to increase the robustness and accuracy of this technique.

Visualization in 2D/3D registration matters for assuring technology-assisted image-guided surgery

PURPOSE

Image-guided navigation and surgical robotics are the next frontiers of minimally invasive surgery. Assuring safety in high-stakes clinical environments is critical for their deployment. 2D/3D registration is an essential, enabling algorithm for most of these systems, as it provides spatial alignment of preoperative data with intraoperative images. While these algorithms have been studied widely, there is a need for verification methods to enable human stakeholders to assess and either approve or reject registration results to ensure safe operation.

METHODS

To address the verification problem from the perspective of human perception, we develop novel visualization paradigms and use a sampling method based on approximate posterior distribution to simulate registration offsets. We then conduct a user study with 22 participants to investigate how different visualization paradigms (Neutral, Attention-Guiding, Correspondence-Suggesting) affect human performance in evaluating the simulated 2D/3D registration results using 12 pelvic fluoroscopy images.

RESULTS

All three visualization paradigms allow users to perform better than random guessing to differentiate between offsets of varying magnitude. The novel paradigms show better performance than the neutral paradigm when using an absolute threshold to differentiate acceptable and unacceptable registrations (highest accuracy: Correspondence-Suggesting (65.1%), highest F1 score: Attention-Guiding (65.7%)), as well as when using a paradigm-specific threshold for the same discrimination (highest accuracy: Attention-Guiding (70.4%), highest F1 score: Corresponding-Suggesting (65.0%)).

CONCLUSION

This study demonstrates that visualization paradigms do affect the human-based assessment of 2D/3D registration errors. However, further exploration is needed to understand this effect better and develop more effective methods to assure accuracy. This research serves as a crucial step toward enhanced surgical autonomy and safety assurance in technology-assisted image-guided surgery.

Twin-S: a digital twin for skull base surgery

PURPOSE

Digital twins are virtual replicas of real-world objects and processes, and they have potential applications in the field of surgical procedures, such as enhancing situational awareness. We introduce Twin-S, a digital twin framework designed specifically for skull base surgeries.

METHODS

Twin-S is a novel framework that combines high-precision optical tracking and real-time simulation, making it possible to integrate it into image-guided interventions. To guarantee accurate representation, Twin-S employs calibration routines to ensure that the virtual model precisely reflects all real-world processes. Twin-S models and tracks key elements of skull base surgery, including surgical tools, patient anatomy, and surgical cameras. Importantly, Twin-S mirrors real-world drilling and updates the virtual model at frame rate of 28.

RESULTS

Our evaluation of Twin-S demonstrates its accuracy, with an average error of 1.39 mm during the drilling process. Our study also highlights the benefits of Twin-S, such as its ability to provide augmented surgical views derived from the continuously updated virtual model, thus offering additional situational awareness to the surgeon.

CONCLUSION

We present Twin-S, a digital twin environment for skull base surgery. Twin-S captures the real-world surgical progresses and updates the virtual model in real time through the use of modern tracking technologies. Future research that integrates vision-based techniques could further increase the accuracy of Twin-S.

Simultaneous Online Registration-Independent Stiffness Identification and Tip Localization of Surgical Instruments in Robot-assisted Eye Surgery

Notable challenges during retinal surgery lend themselves to robotic assistance which has proven beneficial in providing a safe steady-hand manipulation. Efficient assistance from the robots heavily relies on accurate sensing of surgery states (e.g. instrument tip localization and tool-to-tissue interaction forces). Many of the existing tool tip localization methods require preoperative frame registrations or instrument calibrations. In this study using an iterative approach and by combining vision and force-based methods, we develop calibration- and registration-independent (RI) algorithms to provide online estimates of instrument stiffness (least squares and adaptive). The estimations are then combined with a state-space model based on the forward kinematics (FWK) of the Steady-Hand Eye Robot (SHER) and Fiber Brag Grating (FBG) sensor measurements. This is accomplished using a Kalman Filtering (KF) approach to improve the deflected instrument tip position estimations during robot-assisted eye surgery. The conducted experiments demonstrate that when the online RI stiffness estimations are used, the instrument tip localization results surpass those obtained from pre-operative offline calibrations for stiffness.

Flexible Needle Bending Model for Spinal Injection Procedures

An in situ needle manipulation technique used by physicians when performing spinal injections is modeled to study its effect on needle shape and needle tip position. A mechanics-based model is proposed and solved using finite element method. A test setup is presented to mimic the needle manipulation motion. Tissue phantoms made from plastisol as well as porcine skeletal muscle samples are used to evaluate the model accuracy against medical images. The effect of different compression models as well as model parameters on model accuracy is studied, and the effect of needle-tissue interaction on the needle remote center of motion is examined. With the correct combination of compression model and model parameters, the model simulation is able to predict needle tip position within submillimeter accuracy.

Huế dental students' use and perception of an online dental learning platform: A pilot study

PURPOSE/OBJECTIVES

Online educational materials are growing in use, and dental students worldwide can benefit from higher quality and more accessible online supplemental resources. This study was created to evaluate the learning resources non-English speaking dental students desire and to pilot My Dental Key (MDK), an English, evidence-based, online dental educational platform.

METHODS

Third to sixth year dental students at the Huế University of Medicine and Pharmacy were asked to pilot MDK over a 5-week period and were invited to answer three surveys throughout the study. A preliminary survey was given to gauge the participants' (n = 209) preferences regarding the use of English-based dental educational resources. Participants (n = 58) completed a presurvey prior to accessing MDK. After the 5-week period, participants (n = 38) were given a postsurvey to evaluate the platform's effectiveness as a supplemental educational resource.

RESULTS

Overall, we found that: (1) students desire credible online supplemental resources in addition to current resources provided by their school, (2) the multimodal content that MDK provides is a strength that bridges language barriers (3) participants perceived that the content on MDK would help them in class and when treating patients.

CONCLUSIONS

Improving the quality of online supplemental dental resources will have the capability to progress the current educational landscape, and further resources should be created to best serve the global dental community.

Real-time intraoperative surgical guidance system in the da Vinci surgical robot based on transrectal ultrasound/photoacoustic imaging with photoacoustic markers: an ex vivo demonstration

This paper introduces the first integrated real-time intraoperative surgical guidance system, in which an endoscope camera of da Vinci surgical robot and a transrectal ultrasound (TRUS) transducer are co-registered using photoacoustic markers that are detected in both fluorescence (FL) and photoacoustic (PA) imaging. The co-registered system enables the TRUS transducer to track the laser spot illuminated by a pulsed-laser-diode attached to the surgical instrument, providing both FL and PA images of the surgical region-of-interest (ROI). As a result, the generated photoacoustic marker is visualized and localized in the da Vinci endoscopic FL images, and the corresponding tracking can be conducted by rotating the TRUS transducer to display the PA image of the marker. A quantitative evaluation revealed that the average registration and tracking errors were 0.84 mm and 1.16°, respectively. This study shows that the co-registered photoacoustic marker tracking can be effectively deployed intraoperatively using TRUS+PA imaging providing functional guidance of the surgical ROI.

Synthetic data accelerates the development of generalizable learning-based algorithms for X-ray image analysis

Artificial intelligence (AI) now enables automated interpretation of medical images. However, AI's potential use for interventional image analysis remains largely untapped. This is because the post hoc analysis of data collected during live procedures has fundamental and practical limitations, including ethical considerations, expense, scalability, data integrity and a lack of ground truth. Here we demonstrate that creating realistic simulated images from human models is a viable alternative and complement to large-scale in situ data collection. We show that training AI image analysis models on realistically synthesized data, combined with contemporary domain generalization techniques, results in machine learning models that on real data perform comparably to models trained on a precisely matched real data training set. We find that our model transfer paradigm for X-ray image analysis, which we refer to as SyntheX, can even outperform real-data-trained models due to the effectiveness of training on a larger dataset. SyntheX provides an opportunity to markedly accelerate the conception, design and evaluation of X-ray-based intelligent systems. In addition, SyntheX provides the opportunity to test novel instrumentation, design complementary surgical approaches, and envision novel techniques that improve outcomes, save time or mitigate human error, free from the ethical and practical considerations of live human data collection.

A Surgical Robotic System for Osteoporotic Hip Augmentation: System Development and Experimental Evaluation

Minimally-invasive Osteoporotic Hip Augmentation (OHA) by injecting bone cement is a potential treatment option to reduce the risk of hip fracture. This treatment can significantly benefit from computer-assisted planning and execution system to optimize the pattern of cement injection. We present a novel robotic system for the execution of OHA that consists of a 6-DOF robotic arm and integrated drilling and injection component. The minimally-invasive procedure is performed by registering the robot and preoperative images to the surgical scene using multiview image-based 2D/3D registration with no external fiducial attached to the body. The performance of the system is evaluated through experimental sawbone studies as well as cadaveric experiments with intact soft tissues. In the cadaver experiments, distance errors of 3.28mm and 2.64mm for entry and target points and orientation error of 2.30° are calculated. Moreover, the mean surface distance error of 2.13mm with translational error of 4.47mm is reported between injected and planned cement profiles. The experimental results demonstrate the first application of the proposed Robot-Assisted combined Drilling and Injection System (RADIS), incorporating biomechanical planning and intraoperative fiducial-less 2D/3D registration on human cadavers with intact soft tissues.

Mixed Reality Interfaces for Achieving Desired Views with Robotic X-ray Systems

Robotic X-ray C-arm imaging systems can precisely achieve any position and orientation relative to the patient. Informing the system, however, what pose exactly corresponds to a desired view is challenging. Currently these systems are operated by the surgeon using joysticks, but this interaction paradigm is not necessarily effective because users may be unable to efficiently actuate more than a single axis of the system simultaneously. Moreover, novel robotic imaging systems, such as the Brainlab Loop-X, allow for independent source and detector movements, adding even more complexity. To address this challenge, we consider complementary interfaces for the surgeon to command robotic X-ray systems effectively. Specifically, we consider three interaction paradigms: (1) the use of a pointer to specify the principal ray of the desired view relative to the anatomy, (2) the same pointer, but combined with a mixed reality environment to synchronously render digitally reconstructed radiographs from the tool's pose, and (3) the same mixed reality environment but with a virtual X-ray source instead of the pointer. Initial human-in-the-loop evaluation with an attending trauma surgeon indicates that mixed reality interfaces for robotic X-ray system control are promising and may contribute to substantially reducing the number of X-ray images acquired solely during "fluoro hunting" for the desired view or standard plane.

Fluoroscopy-Guided Robotic System for Transforaminal Lumbar Epidural Injections

We present an autonomous robotic spine needle injection system using fluoroscopic image-based navigation. Our system includes patient-specific planning, intra-operative image-based 2D/3D registration and navigation, and automatic robot-guided needle injection. We performed intensive simulation studies to validate the registration accuracy. We achieved a mean spine vertebrae registration error of 0.8 ± 0.3 mm, 0.9 ± 0.7 degrees, mean injection device registration error of 0.2 ± 0.6 mm, 1.2 ± 1.3 degrees, in translation and rotation, respectively. We then conducted cadaveric studies comparing our system to an experienced clinician's free-hand injections. We achieved a mean needle tip translational error of 5.1 ± 2.4 mm and needle orientation error of 3.6 ± 1.9 degrees for robotic injections, compared to 7.6 ± 2.8 mm and 9.9 ± 4.7 degrees for clinician's free-hand injections, respectively. During injections, all needle tips were placed within the defined safety zones for this application. The results suggest the feasibility of using our image-guided robotic injection system for spinal orthopedic applications.

Automated Extraction of Anatomical Measurements From Temporal Bone CT Imaging

OBJECTIVE

Proposed methods of minimally invasive and robot-assisted procedures within the temporal bone require measurements of surgically relevant distances and angles, which often require time-consuming manual segmentation of preoperative imaging. This study aims to describe an automatic segmentation and measurement extraction pipeline of temporal bone cone-beam computed tomography (CT) scans.

STUDY DESIGN

Descriptive study of temporal bone measurements.

SETTING

Academic institution.

METHODS

A propagation template composed of 16 temporal bone CT scans was formed with relevant anatomical structures and landmarks manually segmented. Next, 52 temporal bone CT scans were autonomously segmented using deformable registration techniques from the Advanced Normalization Tools Python package. Anatomical measurements were extracted via in-house Python algorithms. Extracted measurements were compared to ground truth values from manual segmentations.

RESULTS

Paired t test analyses showed no statistical difference between measurements using this pipeline and ground truth measurements from manually segmented images. Mean (SD) malleus manubrium length was 4.39 (0.34) mm. Mean (SD) incus short and long processes were 2.91 (0.18) mm and 3.53 (0.38) mm, respectively. The mean (SD) maximal diameter of the incus long process was 0.74 (0.17) mm. The first and second facial nerve genus had mean (SD) angles of 68.6 (6.7) degrees and 111.1 (5.3) degrees, respectively. The facial recess had a mean (SD) span of 3.21 (0.46) mm. Mean (SD) minimum distance between the external auditory canal and tegmen was 3.79 (1.05) mm.

CONCLUSIONS

This is the first study to automatically extract relevant temporal bone anatomical measurements from CT scans using segmentation propagation. Measurements from these models can streamline preoperative planning, improve future segmentation techniques, and help develop future image-guided or robot-assisted systems for temporal bone procedures.

Force-based Control for Safe Robot-assisted Retinal Interventions: In Vivo Evaluation in Animal Studies

In recent years, robotic assistance in vitreoretinal surgery has moved from a benchtop environment to the operating rooms. Emerging robotic systems improve tool manoeuvrability and provide precise tool motions in a constrained intraocular environment and reduce/remove hand tremor. However, often due to their stiff and bulky mechanical structure, they diminish the perception of tool-to-sclera (scleral) forces, on which the surgeon relies, for eyeball manipulation. In this paper we measure these scleral forces and actively control the robot to keep them under a predefined threshold. Scleral forces are measured using a Fiber Bragg Grating (FBG) based force sensing instrument in an in vivo rabbit eye model in manual, cooperative robotic assistance with no scleral force control (NC), adaptive scleral force norm control (ANC) and adaptive scleral force component control (ACC) methods. To the best of our knowledge, this is the first time that the scleral forces are measured in an in vivo eye model during robot assisted vitreoretinal procedures. An experienced retinal surgeon repeated an intraocular tool manipulation (ITM) task 10 times in four in vivo rabbit eyes and a phantom eyeball, for a total of 50 repetitions in each control mode. Statistical analysis shows that the ANC and ACC control schemes restrict the duration of the undesired scleral forces to 4.41% and 14.53% as compared to 43.30% and 35.28% in manual and NC cases, respectively during the in vivo studies. These results show that the active robot control schemes can maintain applied scleral forces below a desired threshold during robot-assisted vitreoretinal surgery. The scleral forces measurements in this study may enable a better understanding of tool-to-sclera interactions during vitreoretinal surgery and the proposed control strategies could be extended to other microsurgery and robot-assisted interventions.

Bridging 3D Slicer and ROS2 for Image-Guided Robotic Interventions

Developing image-guided robotic systems requires access to flexible, open-source software. For image guidance, the open-source medical imaging platform 3D Slicer is one of the most adopted tools that can be used for research and prototyping. Similarly, for robotics, the open-source middleware suite robot operating system (ROS) is the standard development framework. In the past, there have been several "ad hoc" attempts made to bridge both tools; however, they are all reliant on middleware and custom interfaces. Additionally, none of these attempts have been successful in bridging access to the full suite of tools provided by ROS or 3D Slicer. Therefore, in this paper, we present the SlicerROS2 module, which was designed for the direct use of ROS2 packages and libraries within 3D Slicer. The module was developed to enable real-time visualization of robots, accommodate different robot configurations, and facilitate data transfer in both directions (between ROS and Slicer). We demonstrate the system on multiple robots with different configurations, evaluate the system performance and discuss an image-guided robotic intervention that can be prototyped with this module. This module can serve as a starting point for clinical system development that reduces the need for custom interfaces and time-intensive platform setup.

Statistical Shape Model of the Temporal Bone Using Segmentation Propagation

HYPOTHESIS

Automated image registration techniques can successfully determine anatomical variation in human temporal bones with statistical shape modeling.

BACKGROUND

There is a lack of knowledge about inter-patient anatomical variation in the temporal bone. Statistical shape models (SSMs) provide a powerful method for quantifying variation of anatomical structures in medical images but are time-intensive to manually develop. This study presents SSMs of temporal bone anatomy using automated image-registration techniques.

METHODS

Fifty-three cone-beam temporal bone CTs were included for SSM generation. The malleus, incus, stapes, bony labyrinth, and facial nerve were automatically segmented using 3D Slicer and a template-based segmentation propagation technique. Segmentations were then used to construct SSMs using MATLAB. The first three principal components of each SSM were analyzed to describe shape variation.

RESULTS

Principal component analysis of middle and inner ear structures revealed novel modes of anatomical variation. The first three principal components for the malleus represented variability in manubrium length (mean: 4.47 mm; ±2-SDs: 4.03-5.03 mm) and rotation about its long axis (±2-SDs: -1.6° to 1.8° posteriorly). The facial nerve exhibits variability in first and second genu angles. The bony labyrinth varies in the angle between the posterior and superior canals (mean: 88.9°; ±2-SDs: 83.7°-95.7°) and cochlear orientation (±2-SDs: -4.0° to 3.0° anterolaterally).

CONCLUSIONS

SSMs of temporal bone anatomy can inform surgeons on clinically relevant inter-patient variability. Anatomical variation elucidated by these models can provide novel insight into function and pathophysiology. These models also allow further investigation of anatomical variation based on age, BMI, sex, and geographical location.

Automated Registration-Based Temporal Bone Computed Tomography Segmentation for Applications in Neurotologic Surgery

OBJECTIVE

This study investigates the accuracy of an automated method to rapidly segment relevant temporal bone anatomy from cone beam computed tomography (CT) images. Implementation of this segmentation pipeline has potential to improve surgical safety and decrease operative time by augmenting preoperative planning and interfacing with image-guided robotic surgical systems.

STUDY DESIGN

Descriptive study of predicted segmentations.

SETTING

Academic institution.

METHODS

We have developed a computational pipeline based on the symmetric normalization registration method that predicts segmentations of anatomic structures in temporal bone CT scans using a labeled atlas. To evaluate accuracy, we created a data set by manually labeling relevant anatomic structures (eg, ossicles, labyrinth, facial nerve, external auditory canal, dura) for 16 deidentified high-resolution cone beam temporal bone CT images. Automated segmentations from this pipeline were compared against ground-truth manual segmentations by using modified Hausdorff distances and Dice scores. Runtimes were documented to determine the computational requirements of this method.

RESULTS

Modified Hausdorff distances and Dice scores between predicted and ground-truth labels were as follows: malleus (0.100 ± 0.054 mm; Dice, 0.827 ± 0.068), incus (0.100 ± 0.033 mm; Dice, 0.837 ± 0.068), stapes (0.157 ± 0.048 mm; Dice, 0.358 ± 0.100), labyrinth (0.169 ± 0.100 mm; Dice, 0.838 ± 0.060), and facial nerve (0.522 ± 0.278 mm; Dice, 0.567 ± 0.130). A quad-core 16GB RAM workstation completed this segmentation pipeline in 10 minutes.

CONCLUSIONS

We demonstrated submillimeter accuracy for automated segmentation of temporal bone anatomy when compared against hand-segmented ground truth using our template registration pipeline. This method is not dependent on the training data volume that plagues many complex deep learning models. Favorable runtime and low computational requirements underscore this method's translational potential.

Concepts and Trends n Autonomy for Robot-Assisted Surgery

Surgical robots have been widely adopted with over 4000 robots being used in practice daily. However, these are telerobots that are fully controlled by skilled human surgeons. Introducing "surgeon-assist"-some forms of autonomy-has the potential to reduce tedium and increase consistency, analogous to driver-assist functions for lanekeeping, cruise control, and parking. This article examines the scientific and technical backgrounds of robotic autonomy in surgery and some ethical, social, and legal implications. We describe several autonomous surgical tasks that have been automated in laboratory settings, and research concepts and trends.

SAGE: SLAM with Appearance and Geometry Prior for Endoscopy

In endoscopy, many applications (e.g., surgical navigation) would benefit from a real-time method that can simultaneously track the endoscope and reconstruct the dense 3D geometry of the observed anatomy from a monocular endoscopic video. To this end, we develop a Simultaneous Localization and Mapping system by combining the learning-based appearance and optimizable geometry priors and factor graph optimization. The appearance and geometry priors are explicitly learned in an end-to-end differentiable training pipeline to master the task of pair-wise image alignment, one of the core components of the SLAM system. In our experiments, the proposed SLAM system is shown to robustly handle the challenges of texture scarceness and illumination variation that are commonly seen in endoscopy. The system generalizes well to unseen endoscopes and subjects and performs favorably compared with a state-of-the-art feature-based SLAM system. The code repository is available at https://github.com/lppllppl920/SAGE-SLAM.git.

Delta Robot Kinematic Calibration for Precise Robot-Assisted Retinal Surgery

High precision is required for ophthalmic robotic systems. This paper presents the kinematic calibration for the delta robot which is part of the next generation of Steady-Hand Eye Robot (SHER). A linear error model is derived based on geometric error parameters. Two experiments with different ranges of workspace are conducted with laser sensors measuring displacement. The error parameters are identified and applied in the kinematics to compensate for modeling error. To achieve better accuracy, Bernstein polynomials are adopted to fit the error residuals after compensation. After the kinematic calibration process, the error residuals of the delta robot are reduced to satisfy the clinical requirements.

A Dexterous Robotic System for Autonomous Debridement of Osteolytic Bone Lesions in Confined Spaces: Human Cadaver Studies

This article presents a dexterous robotic system for autonomous debridement of osteolytic bone lesions in confined spaces. The proposed system is distinguished from the state-of-the-art orthopedics systems because it combines a rigid-link robot with a continuum manipulator (CM) that enhances reach in difficult-to-access spaces often encountered in surgery. The CM is equipped with flexible debriding instruments and fiber Bragg grating sensors. The surgeon plans on the patient’s preoperative computed tomography and the robotic system performs the task autonomously under the surgeon’s supervision. An optimization-based controller generates control commands on the fly to execute the task while satisfying physical and safety constraints. The system design and controller are discussed and extensive simulation, phantom and human cadaver experiments are carried out to evaluate the performance, workspace, and dexterity in confined spaces. Mean and standard deviation of target placement are 0.5 and 0.18 mm, and the robotic system covers 91% of the workspace behind an acetabular implant in treatment of hip osteolysis, compared to the 54% that is achieved by conventional rigid tools.

A Handheld Steerable Surgical Drill With a Novel Miniaturized Articulated Joint Module for Dexterous Confined-Space Bone Work

OBJECTIVE

Steerable surgical drills have the potential to minimize intraoperative tissue damage to patients. However, due to their large shaft diameters, large bend radii, and small bend angles, existing steerable drills are unsuitable for dexterous operations in confined spaces. This article presents a handheld steerable drill with a 4.5-mm miniaturized tip, capable of abruptly bending up to 65.

METHODS

To achieve a small tip diameter and a large bend angle, we propose a novel articulated joint module composed of a tendon-driven geared rolling joint and a double universal joint for steering the drill shaft and transmitting drilling torques, respectively. We integrate this joint module with a customized compact actuation unit into a handheld device. The integrated handheld steerable drill is slim and lightweight, supporting burdenless, single-handed grips and easy integration into existing surgical procedures.

RESULTS

Experiments and analysis showed the proposed steerable drill has high distal dexterity and is capable to remove target tissues dexterously through a small passage/incision with minimized collateral damage.

CONCLUSION

The results suggest the potential of the proposed miniaturized articulated drill for dexterous bone work in confined spaces.

SIGNIFICANCE

By enhancing distal dexterity and reach for surgeons when dealing with hard bony tissues, the proposed device can potentially minimize surgical invasiveness and thus collateral tissue damage to patients for a better clinical outcome.

Surgical data science - from concepts toward clinical translation

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Volumetric Accuracy Analysis of Virtual Safety Barriers for Cooperative-Control Robotic Mastoidectomy

HYPOTHESIS

Virtual fixtures can be enforced in cooperative-control robotic mastoidectomies with submillimeter accuracy.

BACKGROUND

Otologic procedures are well-suited for robotic assistance due to consistent osseous landmarks. We have previously demonstrated the feasibility of cooperative-control robots (CCRs) for mastoidectomy. CCRs manipulate instruments simultaneously with the surgeon, allowing the surgeon to control instruments with robotic augmentation of motion. CCRs can also enforce virtual fixtures, which are safety barriers that prevent motion into undesired locations. Previous studies have validated the ability of CCRs to allow a novice surgeon to safely complete a cortical mastoidectomy. This study provides objective accuracy data for CCR-imposed safety barriers in cortical mastoidectomies.

METHODS

Temporal bone phantoms were registered to a CCR using preoperative computed tomography (CT) imaging. Virtual fixtures were created using 3D Slicer, with 2D planes placed along the external auditory canal, tegmen, and sigmoid, converging on the antrum. Five mastoidectomies were performed by a novice surgeon, moving the drill to the limit of the barriers. Postoperative CT scans were obtained, and Dice coefficients and Hausdorff distances were calculated.

RESULTS

The average modified Hausdorff distance between drilled bone and the preplanned volume was 0.351 ± 0.093 mm. Compared with the preplanned volume of 0.947 cm3, the mean volume of bone removed was 1.045 cm3 (difference of 0.0982 cm3 or 10.36%), with an average Dice coefficient of 0.741 (range, 0.665-0.802).

CONCLUSIONS

CCR virtual fixtures can be enforced with a high degree of accuracy. Future studies will focus on improving accuracy and developing 3D fixtures around relevant surgical anatomy.

Towards a Clinically Optimized Tilt Mechanism for Bilateral Micromanipulation with Steady-Hand Eye Robot

Cooperative robotic systems for vitreoretinal surgery can enable novel surgical approaches by allowing the surgeon to perform procedures with enhanced stabilization and high accuracy tool movements. This paper presents the optimization and design of a four-bar linkage type tilt mechanism for a novel Steady-Hand Eye Robot (SHER) which can be used equivalently on both, the left and right patient side, during a bilateral approach with two robots. In this optimization, it is desirable to limit the workspace needed for compensation motions that ensure a virtual remote center of motion (V-RCM). The safety space around the patient, the space for the surgeon's hand and maintaining positional accuracy are also included in the optimization. The applicability of the resulting optimized mechanism was confirmed with a design prototype in a representative mock-up of the surgical setting allowing multiple directions of robot approach towards a medical phantom.

Adaptive Control Improves Sclera Force Safety in Robot-Assisted Eye Surgery: A Clinical Study

The integration of robotics into retinal microsurgery leads to a reduction in surgeon perception of tool-to-tissue interaction forces. This blunting of human tactile sensory input, which is due to the inflexible mass and large inertia of the robotic arm as compared to the milli-Newton scale of the interaction forces and fragile tissues during ophthalmic surgery, identifies a potential iatrogenic risk during robotic eye surgery. In this paper, we aim to evaluate two variants of an adaptive force control scheme implemented on the Steady-Hand Eye Robot (SHER) that are intended to mitigate the risk of unsafe scleral forces. The present study enrolled ten retina fellows and ophthalmology residents into a simulated procedure, which simply asked the trainees to follow retinal vessels in a model retina surgery environment. For this purpose, we have developed a force-sensing (equipped with Fiber Bragg Grating (FBG)) instrument to attach to the robot. A piezo-actuated linear stage for creating random lateral motions to the eyeball phantom has been provided to simulate disturbances during surgery. The SHER and all of its dependencies were set up in an operating room in the Wilmer Eye Institute at the Johns Hopkins Hospital. The clinicians conducted robot-assisted experiments with the adaptive controls incorporated as well as freehand manipulations. The results indicate that the Adaptive Norm Control (ANC) method, is able to maintain scleral forces at predetermined safe levels better than even freehand manipulations. Novice clinicians in robot training however, subjectively preferred freehand maneuvers over robotic manipulations. Clinician preferences once highly skilled with the robot is not assessed in this study.

Spotlight-based 3D Instrument Guidance for Retinal Surgery

Retinal surgery is a complex activity that can be challenging for a surgeon to perform effectively and safely. Image guided robot-assisted surgery is one of the promising solutions that bring significant surgical enhancement in treatment outcome and reduce the physical limitations of human surgeons. In this paper, we demonstrate a novel method for 3D guidance of the instrument based on the projection of spotlight in the single microscope images. The spotlight projection mechanism is firstly analyzed and modeled with a projection on both a plane and a sphere surface. To test the feasibility of the proposed method, a light fiber is integrated into the instrument which is driven by the Steady-Hand Eye Robot (SHER). The spot of light is segmented and tracked on a phantom retina using the proposed algorithm. The static calibration and dynamic test results both show that the proposed method can easily archive 0.5 mm of tip-to-surface distance which is within the clinically acceptable accuracy for intraocular visual guidance.

Three dimensional printing of a low-cost middle-ear training model for surgical management of otosclerosis

BACKGROUND

Surgical management of otosclerosis is technically challenging with studies demonstrating that outcomes are commensurate with surgical experience. Moreover, experts apply less force on the ossicular chain during prosthesis placement than their novice counterparts. Given the predicted decreasing patient pool and the rising cost of human temporal bone specimens it has become more challenging for trainees to receive adequate intraoperative or laboratory-based experience in this procedure. As such, there is a need for a low-cost training model for the procedure. Here we describe such a model.

METHODS

A surgical model of the middle ear was designed using computer aided design (CAD) software. The model consists of four components, the superior three dimensional (3D)-printed component representing the external auditory canal, a 90° torsion spring representing the incus, a 3D-printed base with a stapedotomy underlying the torsion spring, and a 3D-printed phone holder to facilitate video-recording of trials and subsequent calculation of the force applied on the modeled incus. Force applied on the incus is calculated based on Hooke's Law from post-trial computer-vision analysis of recorded video following experimental determination of the spring constant of the modeled incus.

RESULTS

The described model was manufactured with a total cost of $56.50. The spring constant was experimentally determined to be 97.0 mN mm/deg, resulting in an ability to detect force applied to the modeled incus across a range of 1.2 to 5200 mN.

CONCLUSIONS

We have created a low-cost middle-ear training model with measurable objective performance outcomes. The range of detectable force exceeds expected values for the task.Level of Evidence: IV.

Towards Safe In Situ Needle Manipulation for Robot Assisted Lumbar Injection in Interventional MRI

Lumbar injection is an image-guided procedure performed manually for diagnosis and treatment of lower back pain and leg pain. Previously, we have developed and verified an MR-Conditional robotic solution to assisting the needle insertion process. Drawing on our clinical experiences, a virtual remote center of motion (RCM) constraint is implemented to enable our robot to mimic a clinician's hand motion to adjust the needle tip position in situ. Force and image data are collected to study the needle behavior in gel phantoms during this motion, and a mechanics-based needle-tissue interaction model is proposed and evaluated to further examine the underlying physics. This work extends the commonly-adopted notion of an RCM for flexible needles, and introduces new motion parameters to describe the needle behavior. The model parameters can be tuned to match the experimental result to sub-millimeter accuracy, and this proposed needle manipulation method presents a safer alternative to laterally translating the needle during in situ needle adjustments.

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

Telerobotic Operation of Intensive Care Unit Ventilators

Since the first reports of a novel coronavirus (SARS-CoV-2) in December 2019, over 33 million people have been infected worldwide and approximately 1 million people worldwide have died from the disease caused by this virus, COVID-19. In the United States alone, there have been approximately 7 million cases and over 200,000 deaths. This outbreak has placed an enormous strain on healthcare systems and workers. Severe cases require hospital care, and 8.5% of patients require mechanical ventilation in an intensive care unit (ICU). One major challenge is the necessity for clinical care personnel to don and doff cumbersome personal protective equipment (PPE) in order to enter an ICU unit to make simple adjustments to ventilator settings. Although future ventilators and other ICU equipment may be controllable remotely through computer networks, the enormous installed base of existing ventilators do not have this capability. This paper reports the development of a simple, low cost telerobotic system that permits adjustment of ventilator settings from outside the ICU. The system consists of a small Cartesian robot capable of operating a ventilator touch screen with camera vision control via a wirelessly connected tablet master device located outside the room. Engineering system tests demonstrated that the open-loop mechanical repeatability of the device was 7.5 mm, and that the average positioning error of the robotic finger under visual servoing control was 5.94 mm. Successful usability tests in a simulated ICU environment were carried out and are reported. In addition to enabling a significant reduction in PPE consumption, the prototype system has been shown in a preliminary evaluation to significantly reduce the total time required for a respiratory therapist to perform typical setting adjustments on a commercial ventilator, including donning and doffing PPE, from 271 to 109 s.

An Active Steering Hand-held Robotic System for Minimally Invasive Orthopaedic Surgery Using a Continuum Manipulator

This paper presents the development and experimental evaluation of an active steering hand-held robotic system for milling and curved drilling in minimally invasive orthopaedic interventions. The system comprises a cable-driven continuum dexterous manipulator (CDM), an actuation unit with a handpiece, and a flexible, rotary cutting tool. Compared to conventional rigid drills, the proposed system enhances dexterity and reach in confined spaces in surgery, while providing direct control to the surgeon with sufficient stability while cutting/milling hard tissue. Of note, for cases that require precise motion, the system is able to be mounted on a positioning robot for additional controllability. A proportional-derivative (PD) controller for regulating drive cable tension is proposed for the stable steering of the CDM during cutting operations. The robotic system is characterized and tested with various tool rotational speeds and cable tensions, demonstrating successful cutting of three-dimensional and curvilinear tool paths in simulated cancellous bone and bone phantom. Material removal rates (MRRs) of up to 571 mm3/s are achieved for stable cutting, demonstrating great improvement over previous related works.