A Review on Advances in Intra-operative Imaging for Surgery and Therapy: Imagining the Operating Room of the Future

Cross-attention learning enables real-time nonuniform rotational distortion correction in OCT

Nonuniform rotational distortion (NURD) correction is vital for endoscopic optical coherence tomography (OCT) imaging and its functional extensions, such as angiography and elastography. Current NURD correction methods require time-consuming feature tracking/registration or cross-correlation calculations and thus sacrifice temporal resolution. Here we propose a cross-attention learning method for the NURD correction in OCT. Our method is inspired by the recent success of the self-attention mechanism in natural language processing and computer vision. By leveraging its ability to model long-range dependencies, we can directly obtain the spatial correlation between OCT A-lines at any distance, thus accelerating the NURD correction. We develop an end-to-end stacked cross-attention network and design three types of optimization constraints. We compare our method with two traditional feature-based methods and a CNN-based method on two publicly-available endoscopic OCT datasets. We further verify the NURD correction performance of our method on 3D stent reconstruction using a home-built endoscopic OCT system. Our method achieves a ∼3 × speedup to real time (26 ± 3 fps), and superior correction performance.

Intraoperative Three-Dimensional Transvaginal Ultrasound for Hysteroscopic Metroplasty: a Controlled Study

This study aims to evaluate the role of intraoperative transvaginal three-dimensional ultrasound (3DUS) during hysteroscopic metroplasty. This is a prospective cohort of consecutive patients with septate uterus undergoing hysteroscopic metroplasty with intraoperative transvaginal 3DUS guidance compared to a historical control group of patients undergoing hysteroscopic metroplasty without 3DUS. We conducted our research in a tertiary care university hospital in Rome, Italy. This study involved nineteen patients undergoing 3DUS-guided hysteroscopic metroplasty for recurrent abortion or infertility compared to 19 age-matched controls undergoing metroplasty without 3DUS guidance. During hysteroscopic metroplasty, 3DUS was performed in the study group when the operator considered the procedure to be completed, according to standards of operative hysteroscopy. If 3DUS diagnosed a residual septum, the procedure was continued until a 3DUS diagnosis of a normal fundus was obtained. The patients were followed with a 3DUS performed 3 months after the procedure. The numbers of complete resections (residual septum absent), suboptimal resections (measurable residual septum of less than 10 mm), and incomplete resections (residual septum > 10 mm) in the intraoperative 3DUS group were compared to the numbers in the control group with no intraoperative 3DUS. At follow-up, measurable residual septa were obtained in 0% of the patients in the 3DUS-guided group versus 26% in the control group (p = 0.04). Residual septa of > 10 mm were obtained in 0% of the 3DUS group versus 10.5% in the control group (p = 0.48). Intraoperative 3DUS reduces the incidence of suboptimal septal resections at hysteroscopic metroplasty.

A multimodal imaging-guided software for access to primate brains

Background

Imaging-guided access to the brain has become a routine procedure for various research and clinical applications, including drug administration, neurophysiological recording, and sampling tissue. Therefore, open-source software is required to handle such datasets in these specific applications.

New methods

Here, we proposed an open-source tool utilizing different imaging modalities for automating the steps to access the brain. This tool provides means for easily calculating the coordination of the area of interest concerning a specific point of entry. The source and documentation are available at this link.

Results

We have used this software for three different applications: electrophysiological recording, drug infusion in the nonhuman primate brain, and guided biopsy procedure in the human brain. We performed a neural recording of two monkeys' prefrontal cortex and inferior temporal cortex using this software in submillimeter resolution. We also applied our procedure for infusion in the putamen and caudate nuclei in both hemispheres of another group of rhesus monkeys with histological proof in one animal. More so, we validated this software in the human subjects that underwent biopsy surgery with the commercial software used in human biopsy surgery.

Comparison with existing methods

Our software uses different imaging modalities by co-registering them. This will provide structural details of the skull and brain tissue. We can calculate each brain region's coordination at the point of entry by re-slicing the images. Atlas-based image segmentation were implemented in our software. Three mentioned applications of our software in neuroscience will be further discussed in this paper.

Conclusion

In our procedure, working with different imaging modalities provides a precise estimation of the specific region in the brain related to the location of implants or stereotaxic frames. There is no limitation to using metal implants in this procedure.

Towards the OR of the future: introducing an adaptive and technology-embracing OR wing layout

PURPOSE

Overageing and climate change cause a need for making processes in the operating room wing (OR wing) more efficient. While many promising technologies are available today, traditional OR wings are not designed for seamlessly integrating these aids. To overcome this discrepancy, we present and motivate multiple ideas on how to transform current architectural design strategies.

METHODS

The presented concepts originate from expert discussions and studies of the available literature, but also from experiences made in the course of daily care delivery. Additionally, a comprehensive evaluation of current and historic OR theatre designs and the problems which are encountered herein has been conducted.

RESULTS

We present three innovative concepts regarding the restructuring of traditional OR wing layouts. To achieve better process optimization, hygiene, and energy efficiency, we propose to divide the OR wing into separate "patient", "procedure" and "staff" zones. For better flexibility regarding perioperative needs and technology integration, we propose to use a hexagon shape combined with reconfigurable walls for designing operating rooms.

CONCLUSION

The concepts presented herein provide a solid foundation for further considerations regarding perioperative process optimization and seamless integration of technology into modern OR wing facilities. We aim at expanding on these results to develop a comprehensive vision for the OR wing of the future.

Nuclear‐medicine probes: where we are and where we are going

Nuclear medicine probes turned into the key for the identification and precise location of sentinel lymph nodes and other occult lesions (i.e., tumors) by using the systemic administration of radiotracers. Intraoperative nuclear probes are key in the surgical management of some malignancies as well as in the determination of positive surgical margins, thus reducing the extent and potential surgery morbidity.

Depending on their application, nuclear probes are classified into two main categories, namely, counting and imaging. Although counting probes present a simple design, are handheld (to be moved rapidly), and provide only acoustic signals when detecting radiation, imaging probes, also known as cameras, are more hardware‐complex and also able to provide images but at the cost of an increased intervention time as displacing the camera has to be done slowly.

This review article begins with an introductory section to highlight the relevance of nuclear‐based probes and their components as well as the main differences between ionization‐ (semiconductor) and scintillation‐based probes. Then, the most significant performance parameters of the probe are reviewed (i.e., sensitivity, contrast, count rate capabilities, shielding, energy, and spatial resolution), as well as the different types of probes based on the target radiation nature, namely: gamma (γ), beta (β) (positron and electron), and Cherenkov. Various available intraoperative nuclear probes are finally compared in terms of performance to discuss the state‐of‐the‐art of nuclear medicine probes.

The manuscript concludes by discussing the ideal probe design and the aspects to be considered when selecting nuclear‐medicine probes.

Use of intraoperative MRI for resection of intracranial tumors: A nationwide analysis of short-term outcomes

OBJECTIVE

Recent evidence supports the use of intraoperative MRI (iMRI) during resection of intracranial tumors due to its demonstrated efficacy and clinical benefit. Though many single-center investigations have been conducted, larger nationwide outcomes have yet to be characterized.

METHODS

We used the American College of Surgeons National Surgical Quality Improvement Program database to examine baseline characteristics and 30-day postoperative outcomes among patients undergoing craniotomy for tumor resection with and without iMRI. Comparisons between outcomes were accomplished after propensity matching using chi-square tests for categorical variables and Welch two-sample t-tests for continuous variables.

RESULTS

A total of 38,003 patients met inclusion criteria. Of this population, 54 (0.1%) received iMRI, while 37,949 (99.9%) did not receive iMRI. After propensity score matching, the resulting groups consisted of an iMRI group (n = 54) and a matched non-iMRI group (n = 54). Procedures involving iMRI were associated with significantly increased operation length compared to those without (p < 0.01). Length of hospital stay was higher in patients without iMRI, with this difference trending towards significance (p = 0.05) in the unmatched comparison. Patients undergoing craniotomy without iMRI had a higher rate of readmission (p = 0.04). There was no significant difference in occurrence of other adverse events between the two patient groups.

CONCLUSION

Despite increasing operative length, iMRI is not associated with higher infection rate and may have a clinical benefit associated with reducing readmissions and a trend towards reducing inpatient length of stay. Additional nationwide analyses including more iMRI patients would provide further insight into the strength of these findings.

Development of a convolutional neural network for the identification and the measurement of the median nerve on ultrasound images acquired at carpal tunnel level

Background

Deep learning applied to ultrasound (US) can provide a feedback to the sonographer about the correct identification of scanned tissues and allows for faster and standardized measurements. The most frequently adopted parameter for US diagnosis of carpal tunnel syndrome is the increasing of the cross-sectional area (CSA) of the median nerve. Our aim was to develop a deep learning algorithm, relying on convolutional neural networks (CNNs), for the localization and segmentation of the median nerve and the automatic measurement of its CSA on US images acquired at the proximal inlet of the carpal tunnel.

Methods

Consecutive patients with rheumatic and musculoskeletal disorders were recruited. Transverse US images were acquired at the carpal tunnel inlet, and the CSA was manually measured. Anatomical variants were registered. The dataset consisted of 246 images (157 for training, 40 for validation, and 49 for testing) from 103 patients each associated with manual annotations of the nerve boundary. A Mask R-CNN, state-of-the-art CNN for image semantic segmentation, was trained on this dataset to accurately localize and segment the median nerve section. To evaluate the performances on the testing set, precision (Prec), recall (Rec), mean average precision (mAP), and Dice similarity coefficient (DSC) were computed. A sub-analysis excluding anatomical variants was performed. The CSA was automatically measured by the algorithm.

Results

The algorithm correctly identified the median nerve in 41/49 images (83.7%) and in 41/43 images (95.3%) excluding anatomical variants. The following metrics were obtained (with and without anatomical variants, respectively): Prec 0.86 ± 0.33 and 0.96 ± 0.18, Rec 0.88 ± 0.33 and 0.98 ± 0.15, mAP 0.88 ± 0.33 and 0.98 ± 0.15, and DSC 0.86 ± 0.19 and 0.88 ± 0.19. The agreement between the algorithm and the sonographer CSA measurements was excellent [ICC 0.97 (0.94–0.98)].

Conclusions

The developed algorithm has shown excellent performances, especially if excluding anatomical variants. Future research should aim at expanding the US image dataset including a wider spectrum of normal anatomy and pathology. This deep learning approach has shown very high potentiality for a fully automatic support for US assessment of carpal tunnel syndrome.

CrossModalNet: exploiting quality preoperative images for multimodal image registration

A long-standing problem in image-guided radiotherapy is that inferior intraoperative images present a difficult problem for automatic registration algorithms. Particularly for digital radiography (DR) and digitally reconstructed radiograph (DRR), the blurred, low-contrast, and noisy DR makes the multimodal registration of DR-DRR challenging. Therefore, we propose a novel CNN-based method called CrossModalNet to exploit the quality preoperative modality (DRR) for handling the limitations of intraoperative images (DR), thereby improving the registration accuracy. The method consists of two parts: DR-DRR contour predictions and contour-based rigid registration. We have designed the CrossModal Attention Module and CrossModal Refine Module to fully exploit the multiscale crossmodal features and implement the crossmodal interactions during the feature encoding and decoding stages. Then, the predicted anatomical contours of DR-DRR are registered by the classic mutual information method. We collected 2486 patient scans to train CrossModalNet and 170 scans to test its performance. The results show that it outperforms the classic and state-of-the-art methods with 95th percentile Hausdorff distance of 5.82 pixels and registration accuracy of 81.2%. The code is available at https://github.com/lc82111/crossModalNet.

Looking for a perfect match: multimodal combinations of Raman spectroscopy for biomedical applications

Raman spectroscopy has shown very promising results in medical diagnostics by providing label-free and highly specific molecular information of pathological tissue ex vivo and in vivo. Nevertheless, the high specificity of Raman spectroscopy comes at a price, i.e., low acquisition rate, no direct access to depth information, and limited sampling areas. However, a similar case regarding advantages and disadvantages can also be made for other highly regarded optical modalities, such as optical coherence tomography, autofluorescence imaging and fluorescence spectroscopy, fluorescence lifetime microscopy, second-harmonic generation, and others. While in these modalities the acquisition speed is significantly higher, they have no or only limited molecular specificity and are only sensitive to a small group of molecules. It can be safely stated that a single modality provides only a limited view on a specific aspect of a biological specimen and cannot assess the entire complexity of a sample. To solve this issue, multimodal optical systems, which combine different optical modalities tailored to a particular need, become more and more common in translational research and will be indispensable diagnostic tools in clinical pathology in the near future. These systems can assess different and partially complementary aspects of a sample and provide a distinct set of independent biomarkers. Here, we want to give an overview on the development of multimodal systems that use RS in combination with other optical modalities to improve the diagnostic performance.

Robot-Assisted Image-Guided Interventions

Image guidance is a common methodology of minimally invasive procedures. Depending on the type of intervention, various imaging modalities are available. Common imaging modalities are computed tomography, magnetic resonance tomography, and ultrasound. Robotic systems have been developed to enable and improve the procedures using these imaging techniques. Spatial and technological constraints limit the development of versatile robotic systems. This paper offers a brief overview of the developments of robotic systems for image-guided interventions since 2015 and includes samples of our current research in this field.

Mesoscopic fluorescence lifetime imaging: Fundamental principles, clinical applications and future directions

Fluorescence lifetime imaging (FLIm) is an optical spectroscopic imaging technique capable of real-time assessments of tissue properties in clinical settings. Label-free FLIm is sensitive to changes in tissue structure and biochemistry resulting from pathological conditions, thus providing optical contrast to identify and monitor the progression of disease. Technical and methodological advances over the last two decades have enabled the development of FLIm instrumentation for real-time, in situ, mesoscopic imaging compatible with standard clinical workflows. Herein, we review the fundamental working principles of mesoscopic FLIm, discuss the technical characteristics of current clinical FLIm instrumentation, highlight the most commonly used analytical methods to interpret fluorescence lifetime data and discuss the recent applications of FLIm in surgical oncology and cardiovascular diagnostics. Finally, we conclude with an outlook on the future directions of clinical FLIm.

Novel Multimodal, Multiscale Imaging System with Augmented Reality

A novel multimodal, multiscale imaging system with augmented reality capability were developed and characterized. The system offers 3D color reflectance imaging, 3D fluorescence imaging, and augmented reality in real time. Multiscale fluorescence imaging was enabled by developing and integrating an in vivo fiber-optic microscope. Real-time ultrasound-fluorescence multimodal imaging used optically tracked fiducial markers for registration. Tomographical data are also incorporated using optically tracked fiducial markers for registration. Furthermore, we characterized system performance and registration accuracy in a benchtop setting. The multiscale fluorescence imaging facilitated assessing the functional status of tissues, extending the minimal resolution of fluorescence imaging to ~17.5 µm. The system achieved a mean of Target Registration error of less than 2 mm for registering fluorescence images to ultrasound images and MRI-based 3D model, which is within clinically acceptable range. The low latency and high frame rate of the prototype system has shown the promise of applying the reported techniques in clinically relevant settings in the future.

Artificial Intelligence for Ultrasound Informative Image Selection of Metacarpal Head Cartilage. A Pilot Study

Objectives: This study aims to develop an automatic deep-learning algorithm, which is based on Convolutional Neural Networks (CNNs), for ultrasound informative-image selection of hyaline cartilage at metacarpal head level. The algorithm performance and that of three beginner sonographers were compared with an expert assessment, which was considered the gold standard.

Methods: The study was divided into two steps. In the first one, an automatic deep-learning algorithm for image selection was developed using 1,600 ultrasound (US) images of the metacarpal head cartilage (MHC) acquired in 40 healthy subjects using a very high-frequency probe (up to 22 MHz). The algorithm task was to identify US images defined informative as they show enough information to fulfill the Outcome Measure in Rheumatology US definition of healthy hyaline cartilage. The algorithm relied on VGG16 CNN, which was fine-tuned to classify US images in informative and non-informative ones. A repeated leave-four-subject out cross-validation was performed using the expert sonographer assessment as gold-standard. In the second step, the expert assessed the algorithm and the beginner sonographers' ability to obtain US informative images of the MHC.

Results: The VGG16 CNN showed excellent performance in the first step, with a mean area (AUC) under the receiver operating characteristic curve, computed among the 10 models obtained from cross-validation, of 0.99 ± 0.01. The model that reached the best AUC on the testing set, which we named “MHC identifier 1,” was then evaluated by the expert sonographer. The agreement between the algorithm, and the expert sonographer was almost perfect [Cohen's kappa: 0.84 (95% confidence interval: 0.71–0.98)], whereas the agreement between the expert and the beginner sonographers using conventional assessment was moderate [Cohen's kappa: 0.63 (95% confidence interval: 0.49–0.76)]. The conventional obtainment of US images by beginner sonographers required 6.0 ± 1.0 min, whereas US videoclip acquisition by a beginner sonographer lasted only 2.0 ± 0.8 min.

Conclusion: This study paves the way for the automatic identification of informative US images for assessing MHC. This may redefine the US reliability in the evaluation of MHC integrity, especially in terms of intrareader reliability and may support beginner sonographers during US training.

Review of Advanced Medical Telerobots

The advent of telerobotic systems has revolutionized various aspects of the industry and human life. This technology is designed to augment human sensorimotor capabilities to extend them beyond natural competence. Classic examples are space and underwater applications when distance and access are the two major physical barriers to be combated with this technology. In modern examples, telerobotic systems have been used in several clinical applications, including teleoperated surgery and telerehabilitation. In this regard, there has been a significant amount of research and development due to the major benefits in terms of medical outcomes. Recently telerobotic systems are combined with advanced artificial intelligence modules to better share the agency with the operator and open new doors of medical automation. In this review paper, we have provided a comprehensive analysis of the literature considering various topologies of telerobotic systems in the medical domain while shedding light on different levels of autonomy for this technology, starting from direct control, going up to command-tracking autonomous telerobots. Existing challenges, including instrumentation, transparency, autonomy, stochastic communication delays, and stability, in addition to the current direction of research related to benefit in telemedicine and medical automation, and future vision of this technology, are discussed in this review paper.