Cardio Navigation: Planning, Simulation, and Augmented Reality in Robotic Assisted Endoscopic Bypass Grafting

Visual extended reality tools in image-guided surgery in urology: a systematic review

PURPOSE

The aim of this systematic review is to assess the clinical implications of employing various Extended Reality (XR) tools for image guidance in urological surgery.

METHODS

In June 2023, a systematic electronic literature search was conducted using the Medline database (via PubMed), Embase (via Ovid), Scopus, and Web of Science. The search strategy was designed based on the PICO (Patients, Intervention, Comparison, Outcome) criteria. Study protocol was registered on PROSPERO (registry number CRD42023449025). We incorporated retrospective and prospective comparative studies, along with single-arm studies, which provided information on the use of XR, Mixed Reality (MR), Augmented Reality (AR), and Virtual Reality (VR) in urological surgical procedures. Studies that were not written in English, non-original investigations, and those involving experimental research on animals or cadavers were excluded from our analysis. The quality assessment of comparative and cohort studies was conducted utilizing the Newcastle-Ottawa scale, whilst for randomized controlled trials (RCTs), the Jadad scale was adopted. The level of evidence for each study was determined based on the guidelines provided by the Oxford Centre for Evidence-Based Medicine.

RESULTS

The initial electronic search yielded 1,803 papers after removing duplicates. Among these, 58 publications underwent a comprehensive review, leading to the inclusion of 40 studies that met the specified criteria for analysis. 11, 20 and 9 studies tested XR on prostate cancer, kidney cancer and miscellaneous, including bladder cancer and lithiasis surgeries, respectively. Focusing on the different technologies 20, 15 and 5 explored the potential of VR, AR and MR. The majority of the included studies (i.e., 22) were prospective non-randomized, whilst 7 and 11 were RCT and retrospective studies respectively. The included studies that revealed how these new tools can be useful both in preoperative and intraoperative setting for a tailored surgical approach.

CONCLUSIONS

AR, VR and MR techniques have emerged as highly effective new tools for image-guided surgery, especially for urologic oncology. Nevertheless, the complete clinical advantages of these innovations are still in the process of evaluation.

Single Evaluation of Use of a Mixed Reality Headset for Intra-Procedural Image-Guidance during a Mock Laparoscopic Myomectomy on an Ex-Vivo Fibroid Model

Uterine fibroids represent the highest prevalence of benign tumors in women, with reports ranging from 4.5% to 68.6%, with a significant bias towards African American women. For uterine fibroids, a significant decision is determining whether fibroids can be successfully removed using minimally invasive (MI) techniques or their removal requires open surgery. Currently, the standard-of-care for intra-procedural visualization for myomectomies is ultrasound, which has low image quality and requires a specially trained assistant. Currently, the state-of-the-art is to obtain a pre-procedural MRI scan of the patient, which can be used for diagnosis and pre-procedural planning. Although proven incredibly useful pre-procedurally, MRI scans are not often used intra-procedurally due to the inconvenient visualization as 2D slices, which are seen on 2D monitors that do not intuitively convey the depth or orientation of the fibroids, as needed to effectively perform myomectomies. To address this limitation, herein, we present the use of a mixed reality headset (i.e., Microsoft HoloLens 2), as a tool for intra-procedural image-guidance during a mock myomectomy of an ex vivo animal uterus. In this work, we created a patient-specific holographic rendering by performing image segmentation of an MRI scan of a custom-made uterine fibroid animal model. A physician qualitatively assessed the usefulness of the renderings for fibroid localization, as compared to the same visualization on a 2D monitor. In conclusion, the use of mixed reality as an intra-procedural image guidance tool for myomectomies was perceived as a better visualization technique that could lead to improvements in MI approaches and make them accessible to patients from lower socioeconomic populations.

Coronary Heart Disease Preoperative Gesture Interactive Diagnostic System Based on Augmented Reality

Coronary heart disease preoperative diagnosis plays an important role in the treatment of vascular interventional surgery. Actually, most doctors are used to diagnosing the position of the vascular stenosis and then empirically estimating vascular stenosis by selective coronary angiography images instead of using mouse, keyboard and computer during preoperative diagnosis. The invasive diagnostic modality is short of intuitive and natural interaction and the results are not accurate enough. Aiming at above problems, the coronary heart disease preoperative gesture interactive diagnostic system based on Augmented Reality is proposed. The system uses Leap Motion Controller to capture hand gesture video sequences and extract the features which that are the position and orientation vector of the gesture motion trajectory and the change of the hand shape. The training planet is determined by K-means algorithm and then the effect of gesture training is improved by multi-features and multi-observation sequences for gesture training. The reusability of gesture is improved by establishing the state transition model. The algorithm efficiency is improved by gesture prejudgment which is used by threshold discriminating before recognition. The integrity of the trajectory is preserved and the gesture motion space is extended by employing space rotation transformation of gesture manipulation plane. Ultimately, the gesture recognition based on SRT-HMM is realized. The diagnosis and measurement of the vascular stenosis are intuitively and naturally realized by operating and measuring the coronary artery model with augmented reality and gesture interaction techniques. All of the gesture recognition experiments show the distinguish ability and generalization ability of the algorithm and gesture interaction experiments prove the availability and reliability of the system.

Feasibility of four-dimensional preoperative simulation for elbow debridement arthroplasty

BACKGROUND

Recent advances in imaging modalities have enabled three-dimensional preoperative simulation. A four-dimensional preoperative simulation system would be useful for debridement arthroplasty of primary degenerative elbow osteoarthritis because it would be able to detect the impingement lesions.

METHODS

We developed a four-dimensional simulation system by adding the anatomical axis to the three-dimensional computed tomography scan data of the affected arm in one position. Eleven patients with primary degenerative elbow osteoarthritis were included. A "two rings" method was used to calculate the flexion-extension axis of the elbow by converting the surface of the trochlea and capitellum into two rings. A four-dimensional simulation movie was created and showed the optimal range of motion and the impingement area requiring excision. To evaluate the reliability of the flexion-extension axis, interobserver and intraobserver reliabilities regarding the assessment of bony overlap volumes were calculated twice for each patient by two authors. Patients were treated by open or arthroscopic debridement arthroplasties. Pre- and postoperative examinations included elbow range of motion measurement, and completion of the patient-rated questionnaire Hand20, Japanese Orthopaedic Association-Japan Elbow Society Elbow Function Score, and the Mayo Elbow Performance Score.

RESULTS

Measurement of the bony overlap volume showed an intraobserver intraclass correlation coefficient of 0.93 and 0.90, and an interobserver intraclass correlation coefficient of 0.94. The mean elbow flexion-extension arc significantly improved from 101° to 125°. The mean Hand20 score significantly improved from 52 to 22. The mean Japanese Orthopaedic Association-Japan Elbow Society Elbow Function Score significantly improved from 67 to 88. The mean Mayo Elbow Performance Score significantly improved from 71 to 91 at the final follow-up evaluation.

CONCLUSION

We showed that four-dimensional, preoperative simulation can be generated by adding the rotation axis to the one-position, three-dimensional computed tomography image of the affected arm. This method is feasible for elbow debridement arthroplasty.

A review of training research and virtual reality simulators for the da Vinci surgical system

PHENOMENON: Virtual reality simulators are the subject of several recent studies of skills training for robot-assisted surgery. Yet no consensus exists regarding what a core skill set comprises or how to measure skill performance. Defining a core skill set and relevant metrics would help surgical educators evaluate different simulators.

APPROACH

This review draws from published research to propose a core technical skill set for using the da Vinci surgeon console. Publications on three commercial simulators were used to evaluate the simulators' content addressing these skills and associated metrics.

FINDINGS

An analysis of published research suggests that a core technical skill set for operating the surgeon console includes bimanual wristed manipulation, camera control, master clutching to manage hand position, use of third instrument arm, activating energy sources, appropriate depth perception, and awareness of forces applied by instruments. Validity studies of three commercial virtual reality simulators for robot-assisted surgery suggest that all three have comparable content and metrics. However, none have comprehensive content and metrics for all core skills. INSIGHTS: Virtual reality simulation remains a promising tool to support skill training for robot-assisted surgery, yet existing commercial simulator content is inadequate for performing and assessing a comprehensive basic skill set. The results of this evaluation help identify opportunities and challenges that exist for future developments in virtual reality simulation for robot-assisted surgery. Specifically, the inclusion of educational experts in the development cycle alongside clinical and technological experts is recommended.

Predicting target vessel location on robot-assisted coronary artery bypass graft using CT to ultrasound registration

PURPOSE

Although robot-assisted coronary artery bypass grafting (RA-CABG) has gained more acceptance worldwide, its success still depends on the surgeon's experience and expertise, and the conversion rate to full sternotomy is in the order of 15%-25%. One of the reasons for conversion is poor pre-operative planning, which is based solely on pre-operative computed tomography (CT) images. In this paper, the authors propose a technique to estimate the global peri-operative displacement of the heart and to predict the intra-operative target vessel location, validated via both an in vitro and a clinical study.

METHODS

As the peri-operative heart migration during RA-CABG has never been reported in the literatures, a simple in vitro validation study was conducted using a heart phantom. To mimic the clinical workflow, a pre-operative CT as well as peri-operative ultrasound images at three different stages in the procedure (Stage(0)-following intubation; Stage(1)-following lung deflation; and Stage(2)-following thoracic insufflation) were acquired during the experiment. Following image acquisition, a rigid-body registration using iterative closest point algorithm with the robust estimator was employed to map the pre-operative stage to each of the peri-operative ones, to estimate the heart migration and predict the peri-operative target vessel location. Moreover, a clinical validation of this technique was conducted using offline patient data, where a Monte Carlo simulation was used to overcome the limitations arising due to the invisibility of the target vessel in the peri-operative ultrasound images.

RESULTS

For the in vitro study, the computed target registration error (TRE) at Stage(0), Stage(1), and Stage(2) was 2.1, 3.3, and 2.6 mm, respectively. According to the offline clinical validation study, the maximum TRE at the left anterior descending (LAD) coronary artery was 4.1 mm at Stage(0), 5.1 mm at Stage(1), and 3.4 mm at Stage(2).

CONCLUSIONS

The authors proposed a method to measure and validate peri-operative shifts of the heart during RA-CABG. In vitro and clinical validation studies were conducted and yielded a TRE in the order of 5 mm for all cases. As the desired clinical accuracy imposed by this procedure is on the order of one intercostal space (10-15 mm), our technique suits the clinical requirements. The authors therefore believe this technique has the potential to improve the pre-operative planning by updating peri-operative migration patterns of the heart and, consequently, will lead to reduced conversion to conventional open thoracic procedures.

Emerging robotic platforms for minimally invasive surgery

Recent technological advances in surgery have resulted in the development of a range of new techniques that have reduced patient trauma, shortened hospitalization, and improved diagnostic accuracy and therapeutic outcome. Despite the many appreciated benefits of minimally invasive surgery (MIS) compared to traditional approaches, there are still significant drawbacks associated with conventional MIS including poor instrument control and ergonomics caused by rigid instrumentation and its associated fulcrum effect. The use of robot assistance has helped to realize the full potential of MIS with improved consistency, safety and accuracy. The development of articulated, precision tools to enhance the surgeon's dexterity has evolved in parallel with advances in imaging and human-robot interaction. This has improved hand-eye coordination and manual precision down to micron scales, with the capability of navigating through complex anatomical pathways. In this review paper, clinical requirements and technical challenges related to the design of robotic platforms for flexible access surgery are discussed. Allied technical approaches and engineering challenges related to instrument design, intraoperative guidance, and intelligent human-robot interaction are reviewed. We also highlight emerging designs and research opportunities in the field by assessing the current limitations and open technical challenges for the wider clinical uptake of robotic platforms in MIS.

Robotic cardiac surgery: advanced minimally invasive technology hindered by barriers to adoption

Robotic cardiac surgery utilizes the most advanced surgical technology to offer patients a minimally invasive alternative to open surgery in the treatment of a broad range of cardiac pathologies. Although robotics may offer substantial benefits to physicians, patients and healthcare institutions, there are important barriers to its adoption that includes inadequate funding, competition from alternate therapies and challenges in training. There is a growing body of evidence to demonstrate the efficacy of robotic cardiac surgery. Technological innovations are improving patient safety and expanding the indications for robotic cardiac surgery beyond the treatment of mitral valve and coronary artery disease. Robotic cardiac surgery is rapidly becoming a feasible, safe and effective option for the definitive treatment of cardiac disease in the context of 21st century challenges to healthcare provision such as diabetes, obesity and an aging population.

Image-guided interventions: technology review and clinical applications

Image-guided interventions are medical procedures that use computer-based systems to provide virtual image overlays to help the physician precisely visualize and target the surgical site. This field has been greatly expanded by the advances in medical imaging and computing power over the past 20 years. This review begins with a historical overview and then describes the component technologies of tracking, registration, visualization, and software. Clinical applications in neurosurgery, orthopedics, and the cardiac and thoracoabdominal areas are discussed, together with a description of an evolving technology named Natural Orifice Transluminal Endoscopic Surgery (NOTES). As the trend toward minimally invasive procedures continues, image-guided interventions will play an important role in enabling new procedures, while improving the accuracy and success of existing approaches. Despite this promise, the role of image-guided systems must be validated by clinical trials facilitated by partnerships between scientists and physicians if this field is to reach its full potential.

Dex-ray: augmented reality neurosurgical navigation with a handheld video probe

OBJECTIVE

We developed an augmented reality system that enables intraoperative image guidance by using 3-dimensional (3D) graphics overlaid on a video stream. We call this system DEX-Ray and report on its development and the initial intraoperative experience in 12 cases.

METHODS

DEX-Ray consists of a tracked handheld probe that integrates a lipstick-size video camera. The camera looks over the probe's tip into the surgical field. The camera's video stream is augmented with coregistered, multimodality 3D graphics and landmarks obtained during neurosurgical planning with 3D workstations. The handheld probe functions as a navigation device to view and point and as an interaction device to adjust the 3D graphics. We tested the system's accuracy in the laboratory and evaluated it intraoperatively with a series of tumor and vascular cases.

RESULTS

DEX-Ray provided accurate and real-time video-based augmented reality display. The system could be seamlessly integrated into the surgical workflow. The see-through effect revealing 3D information below the surgically exposed surface proved to be of significant value, especially during the macroscopic phase of an operation, providing easily understandable structural navigational information. Navigation in deep and narrow surgical corridors was limited by the camera resolution and light sensitivity.

CONCLUSION

The system was perceived as an improved navigational experience because the augmented see-through effect allowed direct understanding of the surgical anatomy beyond the visible surface and direct guidance toward surgical targets.

[Update on computer- and mechatronic-assisted head and neck surgery in Germany]

A consequence of the ongoing advances in medical navigation is the development of so-called mechatronic assistant systems. Up to now, medical navigation had been used only for additional intrasurgical orientation. But improvements in accuracy in imaging and medical navigation can exceed the surgeon's possible manual accuracy of surgical manipulation. In such cases, mechatronic assistant systems can supplement certain surgical procedures in order to obtain the required precision, such as for positioning of implants. The development and possible use of such mechatronic assistant systems in the head and neck, as well as improvements in the accuracy of medical navigation, are the focus of several working groups. For coordinating and adapting the various research projects, different research groups were called to present their current projects and results in the context of ASKRA (working group for skull-base and craniofacial surgery of the German Society of Oto-Rhino-Laryngology, Head and Neck Surgery) workshops at the German Society for Computer- and Robot-Assisted Surgery (CURAC) convention on 14 October 2006 in Hanover. Different projects were presented, with topics including navigated controlled assistant systems for the frontal and lateral skull base, possibilities for sonographic-induced bone measurement, and requirements for high-precision surgery of the skull base.

Intraoperative laparoscope augmentation for port placement and resection planning in minimally invasive liver resection

In recent years, an increasing number of liver tumor indications were treated by minimally invasive laparoscopic resection. Besides the restricted view, two major intraoperative issues in laparoscopic liver resection are the optimal planning of ports as well as the enhanced visualization of (hidden) vessels, which supply the tumorous liver segment and thus need to be divided (e.g., clipped) prior to the resection. We propose an intuitive and precise method to plan the placement of ports. Preoperatively, self-adhesive fiducials are affixed to the patient's skin and a computed tomography (CT) data set is acquired while contrasting the liver vessels. Immediately prior to the intervention, the laparoscope is moved around these fiducials, which are automatically reconstructed to register the patient to its preoperative imaging data set. This enables the simulation of a camera flight through the patient's interior along the laparoscope's or instruments' axes to easily validate potential ports. Intraoperatively, surgeons need to update their surgical planning based on actual patient data after organ deformations mainly caused by application of carbon dioxide pneumoperitoneum. Therefore, preoperative imaging data can hardly be used. Instead, we propose to use an optically tracked mobile C-arm providing cone-beam CT imaging capability intraoperatively. After patient positioning, port placement, and carbon dioxide insufflation, the liver vessels are contrasted and a 3-D volume is reconstructed during patient exhalation. Without any further need for patient registration, the reconstructed volume can be directly augmented on the live laparoscope video, since prior calibration enables both the volume and the laparoscope to be positioned and oriented in the tracking coordinate frame. The augmentation provides the surgeon with advanced visual aid for the localization of veins, arteries, and bile ducts to be divided or sealed.

[The beginnings of robotic surgery--from the roots up to the da Vinci telemanipulator system]

The history of the robotic surgery is only 22 years old. The article gives a short overview regarding the history of robotics, the surgical robots, the da Vinci telemanipulator system and some further commercial and experimental surgical robotic surgical simulation is also emphasized. Robotic surgery has its own place within the following concepts: 1. computer assisted surgery (CAS), 2. computer integrated surgery (CIS), 3. surgical automation, 4. surgical system integration and 5. artificial intelligence (AI). At the end of the paper there are some important sources of informations regarding robotic surgery.

Pediatric Robotic Surgery: Lessons from a Clinical Experience

PURPOSE

Robotic surgery may improve minimally invasive surgery at high magnification by tremor filtration, motion-scaling, and improved dexterity with the provision of a wrist at the end of the robotic instrument.

MATERIALS AND METHODS

We chose the Zeus Microwrist robotic surgical system as more applicable to small children than the competing da Vinci surgical system. We attempted 57 surgical procedures and completed 54.

RESULTS

Completed procedures included Nissen fundoplication (n = 25), cholecystectomy (n = 18), Heller myotomy (n = 2), splenectomy (n = 2), Morgagni hernia repair (n = 2), and single cases of complex pyloroplasty in the chest, bowel resection, left Bochdalek congenital diaphragmatic hernia repair, esophageal atresia and tracheoesophageal fistula repair, and choledochal cyst excision. There were no complications related to the use of the robot. The mean time for the surgeon at the console using the robot was 117 +/- 39 minutes for Nissen fundoplication, and the total operating room time was 250 +/- 60 minutes. Surgeons found dissection, suturing, and knot tying easier than with conventional laparoscopy. None of the surgeons thought the lack of touch feedback (haptics) was crucial.

CONCLUSION

Robotic surgery offers increased dexterity to the pediatric minimally invasive surgeon, but procedures require more time, and there is no defined patient benefit. The fact that robotic surgery digitalizes minimally invasive surgery creates exciting possibilities for training surgeons, planning operations, and performing surgery at great distances from the operator.

Robotic cardiovascular surgery

Cardiovascular surgery has traditionally been performed through a median sternotomy, allowing the surgeon generous access to the heart and surrounding great vessels. Recently, less invasive methods have been developed to allow the surgeon the same amount of dexterity and accessibility to the heart, thus resulting in a paradigm shift in cardiac surgery. Originally, long instruments without pivot points were used, however; with the application of robotic telemanipulation systems that allow for improved dexterity, the surgeon is able to perform cardiac surgery from a distance not previously possible. In this rapidly evolving field, this article reviews the recent history and clinical results of robotics in cardiovascular surgery.

Target Vessel Detection and Coronary Anastomosis Assessment by Intraoperative 12-MHz Ultrasound

PURPOSE

Our aim was to assess whether the left internal mammary artery, left anterior descending artery, and anastomosis could be visualized by intraoperative ultrasound for safe graft harvesting, optimal anastomotic target selection, and quality control.

DESCRIPTION

In 10 patients, the left internal mammary artery, the left anterior descending artery, and the constructed anastomosis were scanned with 12-MHz epicardial ultrasound. Anastomosis quality was assessed on ultrasound and compared with surgeon score.

EVALUATION

All left internal mammary arteries and left anterior descending arteries could be identified, and pathways could be followed on the ultrasound. Plaque and calcifications were detectable. Deviation from initial coronary anastomotic target was necessary in 2 of 10 patients. None of the constructed anastomoses needed revision. On the anastomotic scans, six anastomoses scored satisfactory and four scored good.

CONCLUSIONS

Epicardial ultrasound was able to evaluate vessel characteristics and coronary anastomosis patency. This can lead to correction of surgical technique related problems in the operating room, possibly improving graft patency. Further advancements could make epicardial ultrasound a cost effective standard for anastomotic quality control. Applying it during robotic-assisted bypass surgery could make this procedure appropriate for more patients.

Robotic surgery setup simulation with the integration of inverse-kinematics computation and medical imaging

At present, there are representative robot operation systems such as da Vinci and ZEUS which have realized minimally invasive surgery by the use of dexterous manipulators. In the operating room, medical staff must prepare and set up an environment in which the robot has optimal freedom of motion and its functions can be fully demonstrated for every case. The range of motion in which the robot can reach and be maneuvered is restricted by the fixed point of the trocar site. We have developed a preoperative planning system with the function of volume rendering of medical images and automatic positioning by applying an inverse-kinematics computation of surgical robot. The motion of a surgical robot can be simulated in advance with the intuitive interface and kinematics computation program running in the background of the system. If robotic surgery planning with volume rendering of DICOM images is possible, the discussion of a surgical plan can be directly made just after the diagnosis considering the patient-specific structure. This kind of setup platform would be essential for the future introduction of surgical robotics into an operating room.

Closed-chest, robotically assisted CABG

Total endoscopic coronary artery bypass grafting is a robotic assisted procedure to graft the left internal thoracic artery to the left anterior descending coronary artery without opening the chest. Through four 1-cm port incisions the procedure can be performed on the beating heart using a telemanipulation system and an endoscopic vacuum stabilizer to locally immobilize the heart.