1
|
Schmidt A, Mohareri O, DiMaio S, Yip MC, Salcudean SE. Tracking and mapping in medical computer vision: A review. Med Image Anal 2024; 94:103131. [PMID: 38442528 DOI: 10.1016/j.media.2024.103131] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Revised: 02/08/2024] [Accepted: 02/29/2024] [Indexed: 03/07/2024]
Abstract
As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.
Collapse
Affiliation(s)
- Adam Schmidt
- Department of Electrical and Computer Engineering, University of British Columbia, 2329 West Mall, Vancouver V6T 1Z4, BC, Canada.
| | - Omid Mohareri
- Advanced Research, Intuitive Surgical, 1020 Kifer Rd, Sunnyvale, CA 94086, USA
| | - Simon DiMaio
- Advanced Research, Intuitive Surgical, 1020 Kifer Rd, Sunnyvale, CA 94086, USA
| | - Michael C Yip
- Department of Electrical and Computer Engineering, University of California San Diego, 9500 Gilman Dr, La Jolla, CA 92093, USA
| | - Septimiu E Salcudean
- Department of Electrical and Computer Engineering, University of British Columbia, 2329 West Mall, Vancouver V6T 1Z4, BC, Canada
| |
Collapse
|
2
|
Haouchine N, Dorent R, Juvekar P, Torio E, Wells WM, Kapur T, Golby AJ, Frisken S. Learning Expected Appearances for Intraoperative Registration during Neurosurgery. MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION : MICCAI ... INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION 2023; 14228:227-237. [PMID: 38371724 PMCID: PMC10870253 DOI: 10.1007/978-3-031-43996-4_22] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
We present a novel method for intraoperative patient-to-image registration by learning Expected Appearances. Our method uses preoperative imaging to synthesize patient-specific expected views through a surgical microscope for a predicted range of transformations. Our method estimates the camera pose by minimizing the dissimilarity between the intraoperative 2D view through the optical microscope and the synthesized expected texture. In contrast to conventional methods, our approach transfers the processing tasks to the preoperative stage, reducing thereby the impact of low-resolution, distorted, and noisy intraoperative images, that often degrade the registration accuracy. We applied our method in the context of neuronavigation during brain surgery. We evaluated our approach on synthetic data and on retrospective data from 6 clinical cases. Our method outperformed state-of-the-art methods and achieved accuracies that met current clinical standards.
Collapse
Affiliation(s)
- Nazim Haouchine
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Reuben Dorent
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Parikshit Juvekar
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Erickson Torio
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - William M Wells
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Tina Kapur
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Alexandra J Golby
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Sarah Frisken
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| |
Collapse
|
3
|
Taleb A, Guigou C, Leclerc S, Lalande A, Bozorg Grayeli A. Image-to-Patient Registration in Computer-Assisted Surgery of Head and Neck: State-of-the-Art, Perspectives, and Challenges. J Clin Med 2023; 12:5398. [PMID: 37629441 PMCID: PMC10455300 DOI: 10.3390/jcm12165398] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 08/08/2023] [Accepted: 08/14/2023] [Indexed: 08/27/2023] Open
Abstract
Today, image-guided systems play a significant role in improving the outcome of diagnostic and therapeutic interventions. They provide crucial anatomical information during the procedure to decrease the size and the extent of the approach, to reduce intraoperative complications, and to increase accuracy, repeatability, and safety. Image-to-patient registration is the first step in image-guided procedures. It establishes a correspondence between the patient's preoperative imaging and the intraoperative data. When it comes to the head-and-neck region, the presence of many sensitive structures such as the central nervous system or the neurosensory organs requires a millimetric precision. This review allows evaluating the characteristics and the performances of different registration methods in the head-and-neck region used in the operation room from the perspectives of accuracy, invasiveness, and processing times. Our work led to the conclusion that invasive marker-based methods are still considered as the gold standard of image-to-patient registration. The surface-based methods are recommended for faster procedures and applied on the surface tissues especially around the eyes. In the near future, computer vision technology is expected to enhance these systems by reducing human errors and cognitive load in the operating room.
Collapse
Affiliation(s)
- Ali Taleb
- Team IFTIM, Institute of Molecular Chemistry of University of Burgundy (ICMUB UMR CNRS 6302), Univ. Bourgogne Franche-Comté, 21000 Dijon, France; (C.G.); (S.L.); (A.L.); (A.B.G.)
| | - Caroline Guigou
- Team IFTIM, Institute of Molecular Chemistry of University of Burgundy (ICMUB UMR CNRS 6302), Univ. Bourgogne Franche-Comté, 21000 Dijon, France; (C.G.); (S.L.); (A.L.); (A.B.G.)
- Otolaryngology Department, University Hospital of Dijon, 21000 Dijon, France
| | - Sarah Leclerc
- Team IFTIM, Institute of Molecular Chemistry of University of Burgundy (ICMUB UMR CNRS 6302), Univ. Bourgogne Franche-Comté, 21000 Dijon, France; (C.G.); (S.L.); (A.L.); (A.B.G.)
| | - Alain Lalande
- Team IFTIM, Institute of Molecular Chemistry of University of Burgundy (ICMUB UMR CNRS 6302), Univ. Bourgogne Franche-Comté, 21000 Dijon, France; (C.G.); (S.L.); (A.L.); (A.B.G.)
- Medical Imaging Department, University Hospital of Dijon, 21000 Dijon, France
| | - Alexis Bozorg Grayeli
- Team IFTIM, Institute of Molecular Chemistry of University of Burgundy (ICMUB UMR CNRS 6302), Univ. Bourgogne Franche-Comté, 21000 Dijon, France; (C.G.); (S.L.); (A.L.); (A.B.G.)
- Otolaryngology Department, University Hospital of Dijon, 21000 Dijon, France
| |
Collapse
|
4
|
Haouchine N, Juvekar P, Nercessian M, Wells W, Golby A, Frisken S. Pose Estimation and Non-Rigid Registration for Augmented Reality During Neurosurgery. IEEE Trans Biomed Eng 2022; 69:1310-1317. [PMID: 34543188 PMCID: PMC9007221 DOI: 10.1109/tbme.2021.3113841] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
OBJECTIVE A craniotomy is the removal of a part of the skull to allow surgeons to have access to the brain and treat tumors. When accessing the brain, a tissue deformation occurs and can negatively influence the surgical procedure outcome. In this work, we present a novel Augmented Reality neurosurgical system to superimpose pre-operative 3D meshes derived from MRI onto a view of the brain surface acquired during surgery. METHODS Our method uses cortical vessels as main features to drive a rigid then non-rigid 3D/2D registration. We first use a feature extractor network to produce probability maps that are fed to a pose estimator network to infer the 6-DoF rigid pose. Then, to account for brain deformation, we add a non-rigid refinement step formulated as a Shape-from-Template problem using physics-based constraints that helps propagate the deformation to sub-cortical level and update tumor location. RESULTS We tested our method retrospectively on 6 clinical datasets and obtained low pose error, and showed using synthetic dataset that considerable brain shift compensation and low TRE can be achieved at cortical and sub-cortical levels. CONCLUSION The results show that our solution achieved accuracy below the actual clinical errors demonstrating the feasibility of practical use of our system. SIGNIFICANCE This work shows that we can provide coherent Augmented Reality visualization of 3D cortical vessels observed through the craniotomy using a single camera view and that cortical vessels provide strong features for performing both rigid and non-rigid registration.
Collapse
|
5
|
Koike T, Kin T, Tanaka S, Sato K, Uchida T, Takeda Y, Uchikawa H, Kiyofuji S, Saito T, Takami H, Takayanagi S, Mukasa A, Oyama H, Saito N. Development of a New Image-Guided Neuronavigation System: Mixed-Reality Projection Mapping Is Accurate and Feasible. Oper Neurosurg (Hagerstown) 2021; 21:549-557. [PMID: 34634817 DOI: 10.1093/ons/opab353] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Accepted: 08/02/2021] [Indexed: 11/14/2022] Open
Abstract
BACKGROUND Image-guided systems improve the safety, functional outcome, and overall survival of neurosurgery but require extensive equipment. OBJECTIVE To develop an image-guided surgery system that combines the brain surface photographic texture (BSP-T) captured during surgery with 3-dimensional computer graphics (3DCG) using projection mapping. METHODS Patients who underwent initial surgery with brain tumors were prospectively enrolled. The texture of the 3DCG (3DCG-T) was obtained from 3DCG under similar conditions as those when capturing the brain surface photographs. The position and orientation at the time of 3DCG-T acquisition were used as the reference. The correct position and orientation of the BSP-T were obtained by aligning the BSP-T with the 3DCG-T using normalized mutual information. The BSP-T was combined with and displayed on the 3DCG using projection mapping. This mixed-reality projection mapping (MRPM) was used prospectively in 15 patients (mean age 46.6 yr, 6 males). The difference between the centerlines of surface blood vessels on the BSP-T and 3DCG constituted the target registration error (TRE) and was measured in 16 fields of the craniotomy area. We also measured the time required for image processing. RESULTS The TRE was measured at 158 locations in the 15 patients, with an average of 1.19 ± 0.14 mm (mean ± standard error). The average image processing time was 16.58 min. CONCLUSION Our MRPM method does not require extensive equipment while presenting information of patients' anatomy together with medical images in the same coordinate system. It has the potential to improve patient safety.
Collapse
Affiliation(s)
- Tsukasa Koike
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Taichi Kin
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Shota Tanaka
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Katsuya Sato
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Tatsuya Uchida
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Yasuhiro Takeda
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Hiroki Uchikawa
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Satoshi Kiyofuji
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Toki Saito
- Department of Clinical Information Engineering, The University of Tokyo, Tokyo, Japan
| | - Hirokazu Takami
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | | | - Akitake Mukasa
- Department of Neurosurgery, Kumamoto University, Kumamoto, Japan
| | - Hiroshi Oyama
- Department of Clinical Information Engineering, The University of Tokyo, Tokyo, Japan
| | - Nobuhito Saito
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
6
|
Koike T, Kin T, Tanaka S, Takeda Y, Uchikawa H, Shiode T, Saito T, Takami H, Takayanagi S, Mukasa A, Oyama H, Saito N. Development of Innovative Neurosurgical Operation Support Method Using Mixed-Reality Computer Graphics. World Neurosurg X 2021; 11:100102. [PMID: 33898969 PMCID: PMC8059082 DOI: 10.1016/j.wnsx.2021.100102] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Accepted: 03/06/2021] [Indexed: 12/22/2022] Open
Abstract
Background In neurosurgery, it is important to inspect the spatial correspondence between the preoperative medical image (virtual space), and the intraoperative findings (real space) to improve the safety of the surgery. Navigation systems and related modalities have been reported as methods for matching this correspondence. However, because of the influence of the brain shift accompanying craniotomy, registration accuracy is reduced. In the present study, to overcome these issues, we developed a spatially accurate registration method of medical fusion 3-dimensional computer graphics and the intraoperative brain surface photograph, and its registration accuracy was measured. Methods The subjects included 16 patients with glioma. Nonrigid registration using the landmarks and thin-plate spline methods was performed for the fusion 3-dimensional computer graphics and the intraoperative brain surface photograph, termed mixed-reality computer graphics. Regarding the registration accuracy measurement, the target registration error was measured by two neurosurgeons, with 10 points for each case at the midpoint of the landmarks. Results The number of target registration error measurement points was 160 in the 16 cases. The target registration error was 0.72 ± 0.04 mm. Aligning the intraoperative brain surface photograph and the fusion 3-dimensional computer graphics required ∼10 minutes on average. The average number of landmarks used for alignment was 24.6. Conclusions Mixed-reality computer graphics enabled highly precise spatial alignment between the real space and virtual space. Mixed-reality computer graphics have the potential to improve the safety of the surgery by allowing complementary observation of brain surface photographs and fusion 3-dimensional computer graphics.
Collapse
Key Words
- 2D, 2-Dimensional
- 3D, 3-Dimensional
- 3DCG, 3-Dimensional computer graphics
- AR, Augmented reality
- Brain shift
- CT, Computed tomography
- Computer graphics
- FOV, Field of view
- Glioma
- Landmark
- MRCG, Mixed-reality computer graphics
- MRI, Magnetic resonance imaging
- Mixed-reality
- TE, Echo time
- TR, Repetition time
- Target registration error
- Thin-plate spline
Collapse
Affiliation(s)
- Tsukasa Koike
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Taichi Kin
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
- To whom correspondence should be addressed: Taichi Kin, M.D.
| | - Shota Tanaka
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Yasuhiro Takeda
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Hiroki Uchikawa
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Taketo Shiode
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Toki Saito
- Department of Clinical Information Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Hirokazu Takami
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Shunsaku Takayanagi
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Akitake Mukasa
- Department of Neurosurgery, Graduate School of Medical Sciences, Kumamoto University, Kumamoto, Japan
| | - Hiroshi Oyama
- Department of Clinical Information Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Nobuhito Saito
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
7
|
Haouchine N, Juvekar P, Wells WM, Cotin S, Golby A, Frisken S. Deformation Aware Augmented Reality for Craniotomy using 3D/2D Non-rigid Registration of Cortical Vessels. MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION : MICCAI ... INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION 2020; 12264:735-744. [PMID: 33778818 PMCID: PMC7999185 DOI: 10.1007/978-3-030-59719-1_71] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Intra-operative brain shift is a well-known phenomenon that describes non-rigid deformation of brain tissues due to gravity and loss of cerebrospinal fluid among other phenomena. This has a negative influence on surgical outcome that is often based on pre-operative planning where the brain shift is not considered. We present a novel brain-shift aware Augmented Reality method to align pre-operative 3D data onto the deformed brain surface viewed through a surgical microscope. We formulate our non-rigid registration as a Shape-from-Template problem. A pre-operative 3D wire-like deformable model is registered onto a single 2D image of the cortical vessels, which is automatically segmented. This 3D/2D registration drives the underlying brain structures, such as tumors, and compensates for the brain shift in sub-cortical regions. We evaluated our approach on simulated and real data composed of 6 patients. It achieved good quantitative and qualitative results making it suitable for neurosurgical guidance.
Collapse
Affiliation(s)
- Nazim Haouchine
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Parikshit Juvekar
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - William M Wells
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
- Massachusetts Institute of Technology, Cambdridge, MA, USA
| | | | - Alexandra Golby
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Sarah Frisken
- Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| |
Collapse
|
8
|
Chen F, Müller J, Müller J, Müller J, Böhl E, Kirsch M, Tetzlaff R. Intraoperative motion correction in neurosurgery: a comparison of intensity- and feature-based methods. BIOMED ENG-BIOMED TE 2018; 63:573-578. [PMID: 30240354 DOI: 10.1515/bmt-2017-0188] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2017] [Accepted: 08/30/2018] [Indexed: 11/15/2022]
Abstract
The intraoperative identification of normal and anomalous brain tissue can be disturbed by pulsatile brain motion and movements of the patient and surgery devices. The performance of four motion correction methods are compared in this paper: Two intensity-based, applying optical flow algorithms, and two feature-based, which take corner features into account to track brain motion. The target registration error with manually selected marking points and the temporal standard deviation of intensity were analyzed in the evaluation. The results reveal the potential of the two types of methods.
Collapse
Affiliation(s)
- Fang Chen
- Institute of Circuits and Systems, Faculty of Electrical and Computer Engineering, Technische Universität Dresden, 01062 Dresden, Germany
| | - Jan Müller
- Institute of Circuits and Systems, Faculty of Electrical and Computer Engineering, Technische Universität Dresden, 01062 Dresden, Germany
| | - Jens Müller
- Institute of Circuits and Systems, Faculty of Electrical and Computer Engineering, Technische Universität Dresden, 01062 Dresden, Germany
| | - Juliane Müller
- Carl Gustav Carus Faculty of Medicine, Department of Anesthesiology and Intensive Care Medicine, Clinical Sensoring and Monitoring, Technische Universität Dresden, 01307 Dresden, Germany
| | - Elisa Böhl
- Carl Gustav Carus Faculty of Medicine, Department of Neurosurgery, Technische Universität Dresden, 01307 Dresden, Germany
| | - Matthias Kirsch
- Carl Gustav Carus Faculty of Medicine, Department of Neurosurgery, Technische Universität Dresden, 01307 Dresden, Germany
| | - Ronald Tetzlaff
- Institute of Circuits and Systems, Faculty of Electrical and Computer Engineering, Technische Universität Dresden, 01062 Dresden, Germany
| |
Collapse
|