Three-dimensional multimodal image-guidance for neurosurgery

Cerebellar Transcranial Magnetic Stimulation (TMS) Impairs Visual Working Memory

An increasing body of evidence points to the involvement of the cerebellum in cognition. Specifically, previous studies have shown that the superior and inferior portions of the cerebellum are involved in different verbal working memory (WM) mechanisms as part of two separate cerebro-cerebellar loops for articulatory rehearsal and phonological storage mechanisms. In comparison, our understanding of the involvement of the cerebellum in visual WM remains limited. We have previously shown that performance in verbal WM is disrupted by single-pulse transcranial magnetic stimulation (TMS) of the right superior cerebellum. The present study aimed to expand on this notion by exploring whether the inferior cerebellum is similarly involved in visual WM. Here, we used fMRI-guided, double-pulse TMS to probe the necessity of left superior and left inferior cerebellum in visual WM. We first conducted an fMRI localizer using the Sternberg visual WM task, which yielded targets in left superior and inferior cerebellum. Subsequently, TMS stimulation of these regions at the end of the encoding phase resulted in decreased accuracy in the visual WM task. Differences in the visual WM deficits caused by stimulation of superior and inferior left cerebellum raise the possibility that these regions are involved in different stages of visual WM.

Automated instrument-tracking for 4D video-rate imaging of ophthalmic surgical maneuvers

Intraoperative image-guidance provides enhanced feedback that facilitates surgical decision-making in a wide variety of medical fields and is especially useful when haptic feedback is limited. In these cases, automated instrument-tracking and localization are essential to guide surgical maneuvers and prevent damage to underlying tissue. However, instrument-tracking is challenging and often confounded by variations in the surgical environment, resulting in a trade-off between accuracy and speed. Ophthalmic microsurgery presents additional challenges due to the nonrigid relationship between instrument motion and instrument deformation inside the eye, image field distortion, image artifacts, and bulk motion due to patient movement and physiological tremor. We present an automated instrument-tracking method by leveraging multimodal imaging and deep-learning to dynamically detect surgical instrument positions and re-center imaging fields for 4D video-rate visualization of ophthalmic surgical maneuvers. We are able to achieve resolution-limited tracking accuracy at varying instrument orientations as well as at extreme instrument speeds and image defocus beyond typical use cases. As proof-of-concept, we perform automated instrument-tracking and 4D imaging of a mock surgical task. Here, we apply our methods for specific applications in ophthalmic microsurgery, but the proposed technologies are broadly applicable for intraoperative image-guidance with high speed and accuracy.

Image guidance and improved accuracy of external ventricular drain tip position particularly in patients with small ventricles

OBJECTIVE

External ventricular drain (EVD) insertion is one of the most common emergency neurosurgical procedures. EVDs are traditionally inserted freehand (FH) in an emergency setting, but often result in suboptimal positioning. Image-guided surgery (IGS) is selectively used to assist placement. However, the accuracy and practicality of IGS use is yet to be reported. In this study, the authors set out to assess if IGS is practical and improves the accuracy of EVD placement.

METHODS

Case notes and images obtained in patients who underwent frontal EVD placement were retrospectively reviewed. Ventriculomegaly was determined by the measurement of the Evans index. EVD location was classified as optimal (ipsilateral frontal horn) or suboptimal (any other location). Propensity score matching of the two groups (IGS vs FH) for the Evans index was performed. Data were analyzed for patient age, diagnosis, number of EVDs, and complications. Those without postoperative CT scans were excluded.

RESULTS

A total of 607 patients with 760 EVDs placed were identified; 331 met inclusion criteria. Of these, 287 were inserted FH, and 44 were placed with IGS; 60.6% of all unmatched FH EVDs were optimal compared with 75% of the IGS group (p = 0.067). The IGS group had a significantly smaller Evans index (p < 0.0001). Propensity score matching demonstrated improved optimal position in the IGS group when compared with the matched FH group (75% vs 43.2%, OR 4.6 [1.5-14.6]; p = 0.002). Patients with an Evans index of ≥ 0.36 derived less benefit (75% in IGS vs 66% in FH, p = 0.5), and those with an Evans index < 0.36 derived more benefit (75% in IGS vs 53% in FH, p = 0.024). The overall EVD complication rate was 36% in the FH group versus 18% in the IGS group (p = 0.056). Revision rates were higher in the FH group (p = 0.035), and the operative times were similar (p = 0.69). Long intracranial EVD catheters were associated with tip malposition irrespective of the group.

CONCLUSIONS

Image guidance is practical and improves the accuracy of EVD placement in patients with small ventricles; thus, it should be considered for these patients.

A Filtering Approach for Image-Guided Surgery With a Highly Articulated Surgical Snake Robot

GOAL

The objective of this paper is to introduce a probabilistic filtering approach to estimate the pose and internal shape of a highly flexible surgical snake robot during minimally invasive surgery.

METHODS

Our approach renders a depiction of the robot that is registered to preoperatively reconstructed organ models to produce a 3-D visualization that can be used for surgical feedback. Our filtering method estimates the robot shape using an extended Kalman filter that fuses magnetic tracker data with kinematic models that define the motion of the robot. Using Lie derivative analysis, we show that this estimation problem is observable, and thus, the shape and configuration of the robot can be successfully recovered with a sufficient number of magnetic tracker measurements.

RESULTS

We validate this study with benchtop and in-vivo image-guidance experiments in which the surgical robot was driven along the epicardial surface of a porcine heart.

CONCLUSION

This paper introduces a filtering approach for shape estimation that can be used for image guidance during minimally invasive surgery.

SIGNIFICANCE

The methods being introduced in this paper enable informative image guidance for highly articulated surgical robots, which benefits the advancement of robotic surgery.

Neck tumor dissection improved with 3-dimensional ultrasound image guidance: technical case report

BACKGROUND AND IMPORTANCE

Three-dimensional ultrasound navigation has been performed to assist in resection of cranial and spinal tumors, but to the best of our knowledge, no one has described the use of real-time 3-dimensional ultrasound navigation in the resection of neck tumors beyond biopsy.

CLINICAL PRESENTATION

This case report describes the use of 3-dimensional ultrasonic navigation in assisting with resection of a large neck paraganglioma. The 3-dimensional ultrasonic navigation improved real-time visualization of the carotid arteries, the trachea, and other vital structures.

CONCLUSION

The use of 3-dimensional ultrasound navigation should be considered in aiding resection of large neck tumors because it can allow more efficient and safer tumor resection.

Cortical vessel imaging and visualization for image guided depth electrode insertion

To avoid intracranial hemorrhage during minimally invasive depth electrode insertion without craniotomy for epilepsy surgery, precise in vivo imaging of cortical vessel and relevant rendering methods are critical, and should be used in preoperative planning. In this study, a non-invasive phase contrast MR angiography (PC-MRA) method was chosen for cortical vessel imaging. After image pre-processing (registration and segmentation), three visualization methods were implemented to optimize the vessel imaging and brain tissue rendering for surgical planning. The processed results were evaluated by comparing with intraoperative photographs. The results showed occurrences of missing vessels between imaging and photos (18.3%, 6 cases), but these could be compensated by realistic sulci visualization methods. The results showed 3D texture mapping to be the most suitable cortex visualization method for use in surgical navigation. Based on the methods and evaluations, a new surgical planning system and criteria of usage were developed with input from the surgeons' experience using the prototype system. This system could greatly help reduce the risk of the intracranial hemorrhage during electrode insertion and also avoid potential risks caused by contrast agent injections for contrast enhanced MRA or CTA.

TMS of posterior parietal cortex disrupts visual tactile multisensory integration

Functional neuroimaging studies have implicated a number of brain regions, especially the posterior parietal cortex (PPC), as being potentially important for visual-tactile multisensory integration. However, neuroimaging studies are correlational and do not prove the necessity of a region for the behavioral improvements that are the hallmark of multisensory integration. To remedy this knowledge gap, we interrupted activity in the PPC, near the junction of the anterior intraparietal sulcus and the postcentral sulcus, using MRI-guided transcranial magnetic stimulation (TMS) while subjects localized touches delivered to different fingers. As the touches were delivered, subjects viewed a congruent touch video, an incongruent touch video, or no video. Without TMS, a strong effect of multisensory integration was observed, with significantly better behavioral performance for discrimination of congruent multisensory touch than for unisensory touch alone. Incongruent multisensory touch produced a smaller improvement in behavioral performance. TMS of the PPC eliminated the behavioral advantage of both congruent and incongruent multisensory stimuli, reducing performance to unisensory levels. These results demonstrate a causal role for the PPC in visual-tactile multisensory integration. Taken together with converging evidence from other studies, these results support a model in which the PPC contains a map of space around the hand that receives input from both the visual and somatosensory modalities. Activity in this map is likely to be the neural substrate for visual-tactile multisensory integration.

Automatic localization of anatomical point landmarks for brain image processing algorithms

Many brain image processing algorithms require one or more well-chosen seed points because they need to be initialized close to an optimal solution. Anatomical point landmarks are useful for constructing initial conditions for these algorithms because they tend to be highly-visible and predictably-located points in brain image scans. We introduce an empirical training procedure that locates user-selected anatomical point landmarks within well-defined precisions using image data with different resolutions and MRI weightings. Our approach makes no assumptions on the structural or intensity characteristics of the images and produces results that have no tunable run-time parameters. We demonstrate the procedure using a Java GUI application (LONI ICE) to determine the MRI weighting of brain scans and to locate features in T1-weighted and T2-weighted scans.

Airway segmentation and measurement in CT images

In this paper we describe a methodology for constructing the airways from Cone Beam CT data and representing changes before and after a medical procedure. A seed region is automatically detected for the first CT slice using a heuristic algorithm incorporating morphological filtering. Our approach then extracts relevant contours on 3D slices by using gradient vector flow (GVF) snakes, modified by an edge detection and snake-shifting step. Following this, a 3D model is constructed. We then estimate the volume of the airway based on segmented 3D shape.

Image-to-patient registration techniques in head surgery

Frame-based stereotaxy was developed in neurosurgery at the beginning of the last century, evolving from atlas-based stereotaxy to stereotaxy based on the individual patient's image data. This established method is still in use in neurosurgery and radiotherapy. There have since been two main developments based on this concept: frameless stereotaxy and markerless registration. Frameless stereotactic systems ('navigation systems') replaced the cumbersome stereotactic frame by mechanically and later also optically or magnetically tracked instruments. Stereotaxy based on the individual patient's image data introduced the problem of patient-to-image data registration. The development of navigation systems based on frameless stereotaxy has dramatically increased its use in surgical disciplines other than neurosurgery, but image-guided surgery based on fiducial marker registration needs dedicated imaging for registration purposes, in addition to the diagnostic imaging that might have been performed. Markerless registration techniques can overcome the resulting additional cost and effort, and result in more widespread use of image-guided surgery techniques. In this review paper, the developments that led to today's navigation systems are outlined, and the applications and possibilities of these methods in the field of maxillofacial surgery are presented.

Real-time SPECT and 2D ultrasound image registration

In this paper we present a technique for fully automatic, real-time 3D SPECT (Single Photon Emitting Computed Tomography) and 2D ultrasound image registration. We use this technique in the context of kidney lesion diagnosis. Our registration algorithm allows a physician to perform an ultrasound exam after a SPECT image has been acquired and see in real time the registration of both modalities. An automatic segmentation algorithm has been implemented in order to display in 3D the positions of the acquired US images with respect to the organs.

Image-guidance for surgical procedures

Contemporary imaging modalities can now provide the surgeon with high quality three- and four-dimensional images depicting not only normal anatomy and pathology, but also vascularity and function. A key component of image-guided surgery (IGS) is the ability to register multi-modal pre-operative images to each other and to the patient. The other important component of IGS is the ability to track instruments in real time during the procedure and to display them as part of a realistic model of the operative volume. Stereoscopic, virtual- and augmented-reality techniques have been implemented to enhance the visualization and guidance process. For the most part, IGS relies on the assumption that the pre-operatively acquired images used to guide the surgery accurately represent the morphology of the tissue during the procedure. This assumption may not necessarily be valid, and so intra-operative real-time imaging using interventional MRI, ultrasound, video and electrophysiological recordings are often employed to ameliorate this situation. Although IGS is now in extensive routine clinical use in neurosurgery and is gaining ground in other surgical disciplines, there remain many drawbacks that must be overcome before it can be employed in more general minimally-invasive procedures. This review overviews the roots of IGS in neurosurgery, provides examples of its use outside the brain, discusses the infrastructure required for successful implementation of IGS approaches and outlines the challenges that must be overcome for IGS to advance further.

Imaging auditory hallucinations in schizophrenia

It is increasingly recognized that there are a heterogeneous range of symptoms within the syndrome of schizophrenia and that some of these also occur frequently within other psychiatric conditions. An approach similar to that in neuropsychology, where cases are grouped based on a discrete deficit, or in this case a discrete symptom, rather than a cause or diagnosis, may be useful in exploring the neural correlates of psychotic symptomatology. Functional neuroimaging provides an excellent tool for investigating the in vivo cortical function of patients with schizophrenia. Auditory verbal hallucinations are one of the most commonly occurring psychotic symptoms in schizophrenia; and this paper examines the progress that has been made in utilizing neuroimaging techniques to investigate auditory hallucinations in schizophrenia and review potential implications for treatment and future directions for research.

Cerebellar transcranial magnetic stimulation impairs verbal working memory

Previous functional magnetic resonance imaging and patient studies indicate cerebellar participation in verbal working memory. In particular, event-related functional magnetic resonance imaging showed superior cerebellar activation during the initial encoding phase of the Sternberg task. This study used functional magnetic resonance imaging-guided transcranial magnetic stimulation (TMS) to test whether disruption of the right superior cerebellum (hemispheric lobule VI/Crus I) impairs verbal working memory performance. Single-pulse TMS was administered immediately after letter presentation during the encoding phase on half the trials. Sham TMS and a Motor Control task were included to test for general distraction and nonmemory-related motor effects. Results showed no effects of TMS on accuracy, but reaction times (RTs) on correct trials were significantly increased on TMS relative to non-TMS trials for the Verbal Working Memory and Motor Control tasks. Additional analyses showed that the increased RT was significantly greater for Verbal Working Memory than for the motor task, suggesting that the effect on working memory was not caused by interference with finger responses. Sham TMS did not affect RTs, indicating that the potentially distracting effects of the postencoding click did not contribute to the increase in RT. The observed effects from cerebellar disruption are consistent with proposed cerebrocerebellar involvement in verbal working memory.

Development and applications of a software for Functional Image Registration (FIRE)

Image registration with anatomical modalities, such as CT and MRI, facilitates the anatomical identification and localization in the interpretation of nuclear medicine images that lack anatomical information. The implementation of Functional Image Registration (FIRE), an operating system (OS) and platform independent multimodal image registration software is reported. In order to register the images without an operator's interaction, several automatic algorithms were implemented. These include principal axes matching and maximization of the mutual information methods. The user interface was designed to support the manual registration of the images. Fused images were composed by overlaying one image with the other one transparently, in which the opacity of the overlaid image was interactively controlled. FIRE was successfully applied to many clinical cases for which automatic and/or manual registration was required. An OS and platform independent program for image registration developed in this study will be useful for the clinical application of image registration techniques.

Software for image registration: algorithms, accuracy, efficacy

Image registration is finding increased clinical use both in aiding diagnosis and guiding therapy. There are numerous algorithms for registration, which all involve maximizing a measure of similarity between a transformed floating image and a fixed reference image. The choice of the similarity measure depends, to some extent, on the application. Methods based on the use of the joint intensity histogram have become popular because of their flexibility and robustness. A distinction is made between rigid-body and non-rigid transformations. The latter are needed for inter-subject registration or intra-subject registration in cases where the region of the body of interest is not considered rigid. Non-rigid transformation is normally achieved using a global model of the deformation but can also be defined by a set of locally rigid transformations, each constrained to a small block in the image. There is scope for further research on the incorporation of appropriate constraints, especially for the application of non-rigid transformations to nuclear medicine studies. Most of the initial practical concerns regarding image registration have been overcome and there is increasing availability of commercial software. There are several approaches to the validation of registration software, with validation of non-rigid algorithms being particularly difficult. Studies have demonstrated the accuracy on the order of half a pixel for both intra- and inter-modality registration (typically 2 to 3 mm). Although hardware-based registration has now become possible by using dual-modality instruments, software-based registration will continue to play an important role in nuclear medicine.

An imaging co-registration system using novel non-invasive and non-radioactive external markers

We present a system of image co-registration and its validation in phantom and volunteer studies. The system co-registered images via six novel non-invasive and non-radioactive external markers. The fiducial markers were attached with sponge bases on the skin surface of the phantom and the volunteers in a non-collinear and non-coplanar array. The subjects were scanned with a 1.5-T magnetic resonance (MR) imager using 2D spin-echo T1-weighted (SE) and 3D spoiled gradient recalled pulse sequences (SPGR) and with a positron emission tomography (PET) scanner for transmission imaging (TI) and emission imaging (EI). The sponge bases created radiolucent gaps with good contrast between the fiducial markers and skin surface. They made the markers visible with clear edge boundaries on both PET and MR images. The images to be registered were rescaled, interpolated, reformatted and followed by point-to-point registration for coordinate determination and the estimation of geometrical transformation and fiducial registration errors (FREs) via the fiducial markers. The images formed four matched pairs of inter-modality (SE-TI, SPGR-TI, SE-EI and SPGR-EI) and two pairs of intra-modality (SE-SPGR, TI-EI) imaging for direct co-registration. The parameters for direct co-registration of SE-TI and SPRG-TI were subsequently used as a bridge and applied for indirect co-registration of SE with EI (SE-EI(TI)) and SPGR with EI (SPGR-EI(TI)), respectively. The overall FREs of the phantom were, respectively, 1.50 mm for inter-modality and 1.10 mm for intra-modality direct co-registration. Those of volunteers were, respectively, 1.79 mm for inter-modality and 1.21 mm for intra-modality direct co-registration. For indirect co-registration, the overall FREs of the phantom were 2.53 mm (SE-EI(TI)) and 2.28 (SPGR-EI(TI)) mm; those of volunteers were 2.84 mm (SE-EI(TI)) and 2.81 mm (SPGR-EI(TI)). The errors of direct co-registration were smaller than those of indirect co-registration; the errors of phantom studies, MR-EI and SPGR-PET were smaller than those of the volunteer studies, MR-TI and SE-PET, respectively (all P<0.01, paired-difference test). In conclusion, motion artefacts, imaging blurring and spatial resolution of imaging remained the key factors affecting the accuracy of co-registration. High-accuracy indirect co-registration is provided by using non-invasive and non-radioactive external fiducial markers. The errors were less than 3 mm for both phantom and volunteer studies. The system is applicable for imaging co-registration of inter-modality non-dual imaging, inter-modality multi-tracer imaging and intra-modality multiple parameter images in clinical practice.

On the incorporation of multi-modality image registration into the radiotherapy treatment planning process

A technique is presented that allows the direct use of physiological image sets in the radiation therapy treatment planning process. When fused to the treatment planning CT, physiological image studies may allow one to define physiological tumor subvolumes consisting of areas of possible chronic hypoxia, areas of high perfusion, areas of high diffusion, and areas containing high choline concentrations. These physiological tumor subvolumes could be selectively boosted to increase local control of malignant brain tumors once one has determined which of these physiological tumor subvolumes predicts for local tumor recurrence after conventional radiotherapy. In this technique a user assisted automatic registration technique is used that is based on an analytical estimate for the transformation matrix needed to register two rigid bodies. The only user input needed is three non-collinear points selected based on landmarks in the primary image and the corresponding three points in the secondary image. Since this registration technique uses two sets of at least three user-defined landmark points each of which has some selection error associated with it, the final registration will have an error that depends only on the selection error associated with the point sets. Since physiological image studies are acquired at the same setting as the T1- w MRI their spatial orientation with respect to the T1- w MRI is known. Therefore, the registration of multiple physiological image studies to the treatment planning CT can be accomplished by first correlating them to the T1- w MRI, and in a second step the T1- w MRI is then registered to the treatment planning CT. The desired registration of the physiological image studies to the treatment planning CT is then accomplished by simply composing the appropriate transformation matrices.

HAMMER: hierarchical attribute matching mechanism for elastic registration

A new approach is presented for elastic registration of medical images, and is applied to magnetic resonance images of the brain. Experimental results demonstrate very high accuracy in superposition of images from different subjects. There are two major novelties in the proposed algorithm. First, it uses an attribute vector, i.e., a set of geometric moment invariants (GMIs) that are defined on each voxel in an image and are calculated from the tissue maps, to reflect the underlying anatomy at different scales. The attribute vector, if rich enough, can distinguish between different parts of an image, which helps establish anatomical correspondences in the deformation procedure; it also helps reduce local minima, by reducing ambiguity in potential matches. This is a fundamental deviation of our method, referred to as the hierarchical attribute matching mechanism for elastic registration (HAMMER), from other volumetric deformation methods, which are typically based on maximizing image similarity. Second, in order to avoid being trapped by local minima, i.e., suboptimal poor matches, HAMMER uses a successive approximation of the energy function being optimized by lower dimensional smooth energy functions, which are constructed to have significantly fewer local minima. This is achieved by hierarchically selecting the driving features that have distinct attribute vectors, thus, drastically reducing ambiguity in finding correspondence. A number of experiments demonstrate that the proposed algorithm results in accurate superposition of image data from individuals with significant anatomical differences.

In vivo quantification of retraction deformation modeling for updated image-guidance during neurosurgery

The use of coregistered preoperative anatomical scans to provide navigational information in the operating room has greatly benefited the field of neurosurgery. Nonetheless, it has been widely acknowledged that significant errors between the operating field and the preoperative images are generated as surgery progresses. Quantification of tissue shift can be accomplished with volumetric intraoperative imaging; however, more functional, lower cost alternative solutions to this challenge are desirable. We are developing the strategy of exploiting a computational model driven by sparse data obtained from intraoperative ultrasound and cortical surface tracking to warp preoperative images to reflect the current state of the operating field. This paper presents an initial quantification of the predictive capability of the current model to computationally capture tissue deformation during retraction in the porcine brain. Performance validation is achieved through comparisons of displacement and pressure predictions to experimental measurements obtained from computed tomographic images and pressure sensor recordings. Group results are based upon a generalized set of boundary conditions for four subjects that, on average, account for at least 75% of tissue motion generated during interhemispheric retraction. Individualized boundary conditions can improve the degree of data-model match by 10% or more but warrant further study. Overall, the level of quantitative agreement achieved in these experiments is encouraging for updating preoperative images to reflect tissue deformation resulting from retraction, especially since model improvements are likely as a result of the intraoperative constraints that can be applied through sparse data collection.

Role of the human medial frontal cortex in task switching: a combined fMRI and TMS study

We used event-related functional magnetic resonance imaging (fMRI) to measure brain activity when subjects were performing identical tasks in the context of either a task-set switch or a continuation of earlier performance. The context, i.e., switching or staying with the current task, influenced medial frontal cortical activation; the medial frontal cortex is transiently activated at the time that subjects switch from one way of performing a task to another. Two types of task-set-switching paradigms were investigated. In the response-switching (RS) paradigm, subjects switched between different rules for response selection and had to choose between competing responses. In the visual-switching (VS) paradigm, subjects switched between different rules for stimulus selection and had to choose between competing visual stimuli. The type of conflict, sensory (VS) or motor (RS), involved in switching was critical in determining medial frontal activation. Switching in the RS paradigm was associated with clear blood-oxygenation-level-dependent signal increases ("activations") in three medial frontal areas: the rostral cingulate zone, the caudal cingulate zone, and the presupplementary motor area (pre-SMA). Switching in the VS task was associated with definite activation in just one medial frontal area, a region on the border between the pre-SMA and the SMA. Subsequent to the fMRI session, we used MRI-guided frameless stereotaxic procedures and repetitive transcranial magnetic stimulation (rTMS) to test the importance of the medial frontal activations for task switching. Applying rTMS over the pre-SMA disrupted subsequent RS performance but only when it was applied in the context of a switch. This result shows, first, that the pre-SMA is essential for task switching and second that its essential role is transient and limited to just the time of behavioral switching. The results are consistent with a role for the pre-SMA in selecting between response sets at a superordinate level rather than in selecting individual responses. The effect of the rTMS was not simply due to the tactile and auditory artifacts associated with each pulse; rTMS over several control regions did not selectively disrupt switching. Applying rTMS over the SMA/pre-SMA area activated in the VS paradigm did not disrupt switching. This result, first, confirms the limited importance of the medial frontal cortex for sensory attentional switching. Second, the VS rTMS results suggest that just because an area is activated in two paradigms does not mean that it plays the same essential role in both cases.

3-D visualization in biomedical applications

Visualizable objects in biology and medicine extend across a vast range of scale, from individual molecules and cells through the varieties of tissue and interstitial interfaces to complete organs, organ systems, and body parts. These objects include functional attributes of these systems, such as biophysical, biomechanical, and physiological properties. Visualization in three dimensions of such objects and their functions is now possible with the advent of high-resolution tomographic scanners and imaging systems. Medical applications include accurate anatomy and function mapping, enhanced diagnosis, accurate treatment planning and rehearsal, and education/training. Biologic applications include study and analysis of structure-to-function relationships in individual cells and organelles. The potential for revolutionary innovation in the practice of medicine and in biologic investigations lies in direct, fully immersive, real-time multisensory fusion of real and virtual information data streams into online, real-time visualizations available during actual clinical procedures or biological experiments. Current high-performance computing, advanced image processing, and high-fidelity rendering capabilities have facilitated major progress toward realization of these goals. With these advances in hand, there are several important applications of three-dimensional visualization that will have a significant impact on the practice of medicine and on biological research.

Mutual information-based rigid and nonrigid registration of ultrasound volumes

We investigated the registration of ultrasound volumes based on the mutual information measure, a technique originally applied to multimodality registration of brain images. A prerequisite for successful registration is a smooth, quasi-convex mutual information surface with an unambiguous maximum. We discuss the necessary preprocessing to create such a surface for ultrasound volumes. Abdominal and thoracic organs imaged with ultrasound typically move relative to the exterior of the body and are deformable. Consequently, four specific instances of image registration involving progressively generalized transformations were studied: rigid-body, rigid-body + uniform scaling, rigid-body + nonuniform scaling, and affine. Registration was applied to clinically acquired volumetric images. The accuracy was comparable with the voxel dimension for all transformation modes, although it degraded as the transformation grew more complex. Likewise, the capture range became narrower with the complexity of transformation. As the use of real-time three-dimensional ultrasound becomes more prevalent, the method we present should work well for a variety of applications examining serial anatomic and physiologic changes. Developers of these clinical applications would match the deformation model of their problem to one of the four transformation models presented here.

Image-guided surgery: from X-rays to virtual reality

Since the discovery of X-rays, medical imaging has played a major role in the guidance of surgical procedures. While medical imaging began with simple X-ray plates to indicate the presence of foreign objects within the human body, the advent of the computer has been a major factor in the recent development of this field. Imaging techniques have grown greatly in their sophistication and can now provide the surgeon with high quality three-dimensional images depicting not only the normal anatomy and pathology, but also vascularity and function. One key factor in the advances in Image-Guided Surgery (IGS) is the ability not only to register images derived from the various imaging modalities amongst themselves, but also to register them to the patient. The other crucial aspect of IGS is the ability to track instruments in real time during the procedure, and to portray them as part of a realistic model of the operative volume. Stereoscopic and virtual-reality techniques can usefully enhance the visualization process. IGS nevertheless relies heavily on the assumption that the images acquired prior to surgery, and upon which the surgical guidance is based, accurately represent the morphology of the tissue during the surgical procedure. In many instances this assumption is invalid, and intra-operative real-time imaging, using interventional MRI, Ultrasound, and electrophysiological recordings are often employed to overcome this limitation. Although now in extensive clinical use, IGS is often currently perceived as an intrusion into the operating room. It must evolve towards becoming a routine surgical tool, but this will only happen if natural and intuitive human interfaces are developed for these systems.

An integrated visualization system for surgical planning and guidance using image fusion and an open MR

A surgical guidance and visualization system is presented, which uniquely integrates capabilities for data analysis and on-line interventional guidance into the setting of interventional MRI. Various pre-operative scans (T1- and T2-weighted MRI, MR angiography, and functional MRI (fMRI)) are fused and automatically aligned with the operating field of the interventional MR system. Both pre-surgical and intra-operative data may be segmented to generate three-dimensional surface models of key anatomical and functional structures. Models are combined in a three-dimensional scene along with reformatted slices that are driven by a tracked surgical device. Thus, pre-operative data augments interventional imaging to expedite tissue characterization and precise localization and targeting. As the surgery progresses, and anatomical changes subsequently reduce the relevance of pre-operative data, interventional data is refreshed for software navigation in true real time. The system has been applied in 45 neurosurgical cases and found to have beneficial utility for planning and guidance. J. Magn. Reson. Imaging 2001;13:967-975.

Symbolic description of intracerebral vessels segmented from magnetic resonance angiograms and evaluation by comparison with X-ray angiograms

We describe and evaluate methods that create detailed vessel trees by linking vessels that have been segmented from magnetic resonance angiograms (MRA). The tree-definition process can automatically exclude erroneous vessel segmentations. The parent-child connectivity information provided by our vessel trees is important to both surgical planning and to guidance of endovascular procedures. We evaluated the branch connection accuracy of our 3D vessel trees by asking two neuroradiologists to evaluate 140 parent-child connections comprising seven vascular trees against 17 digital subtraction angiography (DSA) views. Each reviewer rated each connection as (1) Correct, (2) Incorrect, (3) Partially correct (a minor error without clinical significance), or (4) Indeterminate. Analysis was summarized for each evaluator by calculating 95% confidence intervals for both the proportion completely correct and the proportion clinically acceptable (completely or partially correct). In order to protect the overall Type I error rate, alpha-splitting was done using a top down strategy. We additionally evaluated segmentation completeness by examining each slice in 11 MRA datasets in order to determine unlabeled vessels identifiable in cross-section following segmentation. Results indicate that only one vascular parent-child connection was judged incorrect by both reviewers. MRA segmentations appeared complete within MRA resolution limits. We conclude that our methods permit creation of detailed vascular trees from segmented 3D image data. We review the literature and compare other approaches to our own. We provide examples of clinically useful visualizations enabled by our methodology and taken from a visualization program now in clinical use.

High-Dose Chemotherapy with Autologous Stem Cell Rescue as a Treatment Modality for Breast Cancer

Retrospective analyses of clinical trials as well as several prospective randomized clinical trials designed to evaluate dose intensity have emphasized the importance of intensive chemotherapy in determining outcomes in breast cancer treatment. One strategy that can deliver a large increment in dose intensity is high-dose chemotherapy with autologous stem cell transplantation (HDC-ASCT), since it overcomes myelosuppression, a major dose-limiting toxicity of most chemotherapy regimens. Phase II data from HDC-ASCT trials have indicated improved complete remission rates and longer remission duration when compared to historical conventional-dose chemotherapy regimens. With declining treatment-related mortality (TRM) through improvements in supportive care and innovations in stem cell collection procedures, exploration of HDC-ASCT in prospective randomized trials has been made possible. Several prospective randomized trials have completed accrual and early results are now available, while other studies are still ongoing. Unfortunately the trial designs are quite varied, the characteristics of patients differ from study to study, the early data provide conflicting results, and the follow-up is quite short, making firm conclusions impossible at present. Of note, the comparison arms in several of the studies are performing better than historical nontransplant treatments, perhaps a result of an intermediate dose intensity employed in the nontransplant arm of these trials. These data suggest that dose intensity, whether supported by stem cells or hematopoietic growth factors, may be quite important in optimizing treatment outcomes, affirming earlier concepts of the importance of dose intensity. Further follow-up of the current clinical trials and analyses of ongoing studies not yet reported are necessary before firm conclusions are possible as to the optimal strategy of using dose intensity in the management of breast cancer.

3D reconstruction of the encapsulating contour of arteriovenous malformations for radiosurgery using digital subtraction angiography

PURPOSE

Treatment planning for radiosurgery depends on the precise definition of radiation target volumes. For vascular pathologies such as arteriovenous malformations (AVM), the most usual technique remains standard X-ray projection imaging, most often carried out under stereotactic conditions. To further benefit from the advantages of two-dimensional digital subtraction angiography (DSA), the authors have developed a method for determining the three-dimensional shape of arteriovenous malformations from two views.

METHODS AND MATERIALS

After correction of image intensifier distortion and calibration of both views, the 3D shape of the AVM was determined from two DSA projections using epipolarity geometry. The AVM-encapsulating contour was modeled by triangulation of a stack of almost parallel ellipses. The method was technically validated using artificial targets in a skull phantom. Clinical validation was carried out on 10 patients who were examined using both conventional angiography under stereotactic conditions (SX-ray) and DSA.

RESULTS

There was excellent agreement between the artificial target volumes measured with SX-ray and with DSA. The correspondence between AVM volumes found for patients was not as good as with the phantom.

CONCLUSIONS

The different image characteristics of the two modalities lead to some differences in AVM estimations. However, the results were sufficiently satisfactory to justify routine use of this AVM modeling technique for radiosurgery planning.

In Vivo Analysis of Heterogeneous Brain Deformation Computations for Model-Updated Image Guidance

Neurosurgical image-guidance has historically relied on the registration of the patient and preoperative imaging series with surgical instruments in the operating room (OR) coordinate space. Recent studies measuring intraoperative tissue motion have suggested that deformation-induced misregistration from surgical loading is a serious concern with such systems. In an effort to improve registration fidelity during surgery, we are pursuing an approach which uses a predictive computational model in conjunction with data available in the OR to update the high resolution preoperative image series. In previous work, we have developed an in vivo experimental system in the porcine brain which has been used to investigate a homogeneous finite element rendering of consolidation theory as a tissue deformation model. In this paper, our computational approach has been extended to include heterogeneous tissue property distributions determined from an image-to-grid segmentation scheme. Results produced under two different loading conditions show that heterogeneity in the stiffness properties and interstitial pressure gradients varied over a range of physiologically reasonable values account for 1-3% and 5-8% of the predicted tissue motion, respectively, while homogeneous linear elasticity is responsible for 60-70% of the surgically-induced motion that has been recoverable with our model-based approach.

Depth-buffer targeting for spatially accurate 3-D visualization of medical images

During interactive image-guided surgery (IIGS), a surgeon uses data from medical images to help guide the surgical procedure. At Vanderbilt University, an IIGS software system called Orion has been developed which is capable of displaying up to four 512 x 512 images and the current surgical position using an active optical tracking system. Orion is capable of displaying data from any tomographic image volume and from any NTSC video image. An additional display module has been implemented to display three-dimensional information as well as the tomographic slices. This provides the surgeon with valuable anatomical information that is not readily obtained from the tomographic slices alone. Before the surgery, a set of rendered images is created, each with a different angular view of the tomographic volume in order to surround the site of surgical interest. The major objectives of the display module are to display the appropriate rendered image from the set, identify the current probe position on the selected image, and provide an indication of distance between the probe and the physical point of the anatomy indicated on the image. This can provide the surgeon with vital information such as distance to blood vessels, tumors, or other critical structures.

Fusion of coregistered cross-modality images using a temporally alternating display method

As an aid to the interpretation of functional images, cross-modality coregistration of functional and anatomical images has grown rapidly. Various ways of easily interpreting and visualising coregistered images have previously been investigated; for their display, an intensity-weighted temporally alternating method is used. For brain images, geometric registration involves the automatic alignment method, using the head scalp boundary extracted from the sinogram of a PET emission scan and a surface-matching algorithm; images of the chest or abdomen are registered semi-automatically using a paired point matching algorithm. For the simultaneous display of geometrically registered images, rapid image switching is applied; both images are written with independent colour scales. The rapidly alternating display of two images, synchronised with monitor scanning, induces the fusion of images in the human visual perception system. The accuracy of registration of PET and MRI images is within 2 mm for two point sets. A resulting image is intensified by weighting the display time and/or controlling the intensity map of each image with the degree of interest. This method may be useful for the interpretation and visualisation of coregistered images.

In vivo quantification of a homogeneous brain deformation model for updating preoperative images during surgery

Clinicians using image-guidance for neurosurgical procedures have recently recognized that intraoperative deformation from surgical loading can compromise the accuracy of patient registration in the operating room. While whole brain intraoperative imaging is conceptually appealing it presents significant practical limitations. Alternatively, a promising approach may be to combine incomplete intraoperatively acquired data with a computational model of brain deformation to update high resolution preoperative images during surgery. The success of such an approach is critically dependent on identifying a valid model of brain deformation physics. Towards this end, we evaluate a three-dimensional finite element consolidation theory model for predicting brain deformation in vivo through a series of controlled repeat-experiments. This database is used to construct an interstitial pressure boundary condition calibration curve which is prospectively tested in a fourth validation experiment. The computational model is found to recover 75%-85% of brain motion occurring under loads comparable to clinical conditions. Additionally, the updating of preoperative images using the model calculations is presented and demonstrates that model-updated image-guided neurosurgery may be a viable option for addressing registration errors related to intraoperative tissue motion.

Localization of somatosensory function by using positron emission tomography scanning: a comparison with intraoperative cortical stimulation

Object

To investigate the utility of [15O]H2O positron emission tomography (PET) activation studies in the presurgical mapping of primary somatosensory cortex, the authors compared the magnitude and location of activation foci obtained using PET scanning with the results of intraoperative cortical stimulation (ICS).

Methods

The authors used PET scanning and vibrotactile stimulation (of the face, hand, or foot) to localize the primary somatosensory cortex before surgical resection of mass lesions or epileptogenic foci affecting the central area in 20 patients. With the aid of image-guided surgical systems, the locations of significant activation foci on PET scanning were compared with those of positive ICS performed at craniotomy after the patient had received a local anesthetic agent. In addition, the relationship between the magnitude and statistical significance of blood flow changes and the presence of positive ICS was examined.

In 22 (95.6%) of 23 statistically significant (p < 0.05) PET activation foci, spatially concordant sites on ICS were also observed. Intraoperative cortical stimulation was positive in 40% of the PET activation studies that did not result in statistically significant activation. In the patients showing these results, there was a clearly identifiable t-statistic peak that was spatially concordant with the site of positive ICS in the sensorimotor area. All PET activation foci with a t statistic greater than 4.75 were associated with spatially concordant sites of positive ICS. All PET activation foci with a t statistic less than 3.2 were associated with negative ICS.

Conclusions

Positron emission tomography is an accurate method for mapping the primary somatosensory cortex before surgery. The need for ICS, which requires local anesthesia, may be eliminated when PET foci with high (> 4.75) or low (< 3.20) t-statistic peaks are elicited by vibrotactile stimulation.

Model-updated image guidance: initial clinical experiences with gravity-induced brain deformation

Image-guided neurosurgery relies on accurate registration of the patient, the preoperative image series, and the surgical instruments in the same coordinate space. Recent clinical reports have documented the magnitude of gravity-induced brain deformation in the operating room and suggest these levels of tissue motion may compromise the integrity of such systems. We are investigating a model-based strategy which exploits the wealth of readily-available preoperative information in conjunction with intraoperatively acquired data to construct and drive a three dimensional (3-D) computational model which estimates volumetric displacements in order to update the neuronavigational image set. Using model calculations, the preoperative image database can be deformed to generate a more accurate representation of the surgical focus during an operation. In this paper, we present a preliminary study of four patients that experienced substantial brain deformation from gravity and correlate cortical shift measurements with model predictions. Additionally, we illustrate our image deforming algorithm and demonstrate that preoperative image resolution is maintained. Results over the four cases show that the brain shifted, on average, 5.7 mm in the direction of gravity and that model predictions could reduce this misregistration error to an average of 1.2 mm.

3D reconstruction of the cerebral arterial network from stereotactic DSA

The authors present an automatic algorithm for 3D reconstruction of cerebral blood vessels by digital subtracted angiography. The patient is localized by a stereotactic method. The reconstruction algorithm includes two steps: first vessel extraction then 2D matching and reconstruction. Accurate vessel skeletons are generated by a combination of mathematical morphological algorithms and adaptive filters. The 3D reconstruction algorithm is based on the reconstruction of vessels center lines. For that purpose, three different projections of the vascular network are used. Reconstruction is computed segment by segment (a curved line between two nodes). For each segment point, the algorithm defines all epipolar solutions on the other views. These epipolar solutions are sorted and pooled by 2D continuity and 3D proximity criteria resulting in a 3D graph. Optimal 3D segment is defined by a recursive algorithm that looks up the better path in the 3D graph. The algorithms have been implemented on a Compatible-PC computer in C language. More than 95% of static copper phantom was reconstructed in 5 min and with 1 mm 3D accuracy. 70% of arteries (from carotid to the seventh node) of a true patient arterial network were reconstructed is less than 30 min.

Modeling and optimization of rotational C-arm stereoscopic X-ray angiography

Stereoscopy can be an effective method for obtaining three-dimensional (3-D) spatial information from two-dimensional (2-D) projection X-ray images, without the need for tomographic reconstruction. This much-needed information is missed in many X-ray diagnostic and interventional procedures, such as the treatment of vascular aneurysms. Fast C-arm X-ray systems can obtain multiple angle sequences of stereoscopic image pairs from a single contrast injection and a single breath hold. To advance this solution, we developed a model of stereo angiography, performed perception experiments and related results to optimal acquisition. The model described horizontal disparity for the C-arm geometry that agreed very well with measurements from a geometric phantom. The perceptual accommodation-convergence conflict and geometry limited the effective stereoscopic field of view (SFOV). For a typical large image intensifier system, it was 28 cm x 31 cm at the center of rotation (COR). In the model, blurring from finite focal-spot size and C-arm motion reduced depth resolution on the digital display. Near the COR, the predicted depth resolution was 3-11 mm for a viewing angle of 7 degrees , which agreed favorably with results from recently published studies. The model also described how acquisition parameters affected spatial warping of curves of equal apparent depth. Pincushioning and the difference between the acquisition and display geometry were found to introduce additional distortions to stereo displays. Preference studies on X-ray angiograms indicated that the ideal viewing angle should be small (1-2 degrees), which agreed with some previously published work. Perceptual studies indicated that stereo angiograms should have high artery contrast and that digital processing to increase contrast improved stereopsis. Digital subtraction angiograms, with different motion errors between the left and right-eye views, gave artifacts that confused stereopsis. The addition of background to subtracted images reduced this effect and provided other features for improved depth perception. Using the modeling results and typical clinical angiography requirements, we recommend acquisition protocols and engineering specifications that are achievable on current high-end systems.

Localization of somatosensory function by using positron emission tomography scanning: a comparison with intraoperative cortical stimulation

OBJECT

To investigate the utility of [15O]H2O positron emission tomography (PET) activation studies in the presurgical mapping of primary somatosensory cortex, the authors compared the magnitude and location of activation foci obtained using PET scanning with the results of intraoperative cortical stimulation (ICS).

METHODS

The authors used PET scanning and vibrotactile stimulation (of the face, hand, or foot) to localize the primary somatosensory cortex before surgical resection of mass lesions or epileptogenic foci affecting the central area in 20 patients. With the aid of image-guided surgical systems, the locations of significant activation foci on PET scanning were compared with those of positive ICS performed at craniotomy after the patient had received a local anesthetic agent. In addition, the relationship between the magnitude and statistical significance of blood flow changes and the presence of positive ICS was examined. In 22 (95.6%) of 23 statistically significant (p < 0.05) PET activation foci, spatially concordant sites on ICS were also observed. Intraoperative cortical stimulation was positive in 40% of the PET activation studies that did not result in statistically significant activation. In the patients showing these results, there was a clearly identifiable t-statistic peak that was spatially concordant with the site of positive ICS in the sensorimotor area. All PET activation foci with a t statistic greater than 4.75 were associated with spatially concordant sites of positive ICS. All PET activation foci with a t statistic less than 3.2 were associated with negative ICS.

CONCLUSIONS

Positron emission tomography is an accurate method for mapping the primary somatosensory cortex before surgery. The need for ICS, which requires local anesthesia, may be eliminated when PET foci with high (> 4.75) or low (< 3.20) t-statistic peaks are elicited by vibrotactile stimulation.

A computational model for tracking subsurface tissue deformation during stereotactic neurosurgery

Recent advances in the field of stereotactic neurosurgery have made it possible to coregister preoperative computed tomography (CT) and magnetic resonance (MR) images with instrument locations in the operating field. However, accounting for intraoperative movement of brain tissue remains a challenging problem. While intraoperative CT and MR scanners record concurrent tissue motion, there is motivation to develop methodologies which would be significantly lower in cost and more widely available. The approach we present is a computational model of brain tissue deformation that could be used in conjunction with a limited amount of concurrently obtained operative data to estimate subsurface tissue motion. Specifically, we report on the initial development of a finite element model of brain tissue adapted from consolidation theory. Validations of the computational mathematics in two and three dimensions are shown with errors of 1%-2% for the discretizations used. Experience with the computational strategy for estimating surgically induced brain tissue motion in vivo is also presented. While the predicted tissue displacements differ from measured values by about 15%, they suggest that exploiting a physics-based computational framework for updating preoperative imaging databases during the course of surgery has considerable merit. However, additional model and computational developments are needed before this approach can become a clinical reality.

Systematic distortions in magnetic position digitizers

Medical devices equipped with position sensors enable applications like image guided surgical interventions, reconstruction of three-dimensional 3D ultrasound (US) images, and virtual or augmented reality systems. The acquisition of three-dimensional position data in real time is one of the key technologies in this field. The systematic distortions induced by various metals, surgical tools, and US scan probes in different commercial electromagnetic tracking systems were assessed in the presented work. A precise nonmetallic six degree-of-freedom measurement rack was built that allowed a quantitative comparison of different electromagnetic trackers. Also, their performance in the presence of large metallic structures was quantified in a phantom study on an acrylic skull model in an operating room (OR). The trackers used were alternating current (ac) and direct current (dc) based systems. The ac trackers were, on average, distorted by 0.7 mm and 0.5 degree by metallic objects positioned at a distance greater than 120 mm between the geometrical center of the sample and the sensor. In the OR environment, the ac system exhibits mean errors of 3.2 +/- 2.4 mm and 2.9 degrees +/- 1.9 degrees. The dc trackers are more sensitive to distortions caused by ferromagnetic materials (averaged value: 1.6 mm and 0.5 degree beyond a distance of 120 mm). The dc tracker shows no distortions from other conductive materials but was less accurate in the OR environment (typical error: 6.4 +/- 2.5 mm and 4.9 degrees +/- 2.0 degrees). At distances smaller than approximately 100 mm between sample and sensor error increases quickly. It is also apparent from our measurements that the influence of US scan probes is governed by their shielding material. The results show that surgical instruments not containing conductive material are to be preferred when using an ac tracker. Nonferromagnetic instruments should be used with dc trackers. Static distortions caused by the OR environment have to be compensated by precise calibration methods.

Automated atlas integration and interactive three-dimensional visualization tools for planning and guidance in functional neurosurgery

Many critical functionally distinct subcortical structures are not distinguishable on anatomical magnetic resonance imaging (MRI) scans. In order to provide the neurosurgeon with this missing information, a deformable volumetric atlas of the basal ganglia and thalamus has been created from the Schaltenbrand and Wahren atlas of cryogenic slices. The volumetric atlas can be automatically deformed to an individual patient's MRI. To facilitate the clinical use of the atlas, a visualization platform has been developed for preoperative and intraoperative use which permits manipulation of the merged atlas and MRI data sets in two- and three-dimensional views. The platform includes graphical tools which allow the visualization of projections of a leukotome and other surgical tools with respect to the atlas data, as well as preregistered images from any other imaging modality. In addition, a graphical interface has been designed to create custom virtual lesions using computer models of neurosurgical tools for intraoperative planning. To date this system has been employed as an adjunct to over 30 functional neurosurgical cases including surgery for movement disorders.

Initial In-Vivo Analysis of 3D Heterogeneous Brain Computations for Model-Updated Image-Guided Neurosurgery

Registration error resulting from intraoperative brain shift due to applied surgical loads has long been recognized as one of the most challenging problems in the field of frameless stereotactic neurosurgery. To address this problem, we have developed a 3-dimensional finite element model of the brain and have begun to quantify its predictive capability in an in vivo porcine model. Previous studies have shown that we can predict the average total displacement within 15% and 6.6% error using intraparenchymal and temporal deformation sources, respectively, under relatively simple model assumptions. In this paper, we present preliminary results using a heterogeneous model with an expanding temporally located mass and show that we are capable of predicting an average total displacement to 5.7% under similar model initial and boundary conditions. We also demonstrate that our approach can be viewed as having the capability of recapturing approximately 75% of the registration inaccuracy that may be generated by preoperative-based image-guided neurosurgery.