Merging virtual objects with the real world

The Impact of Occlusion on Depth Perception at Arm's Length

This paper investigates the accuracy of Augmented Reality (AR) technologies, particularly commercially available optical see-through displays, in depicting virtual content inside the human body for surgical planning. Their inherent limitations result in inaccuracies in perceived object positioning. We examine how occlusion, specifically with opaque surfaces, affects perceived depth of virtual objects at arm's length working distances. A custom apparatus with a half-silvered mirror was developed, providing accurate depth cues excluding occlusion, differing from commercial displays. We carried out a study, contrasting our apparatus with a HoloLens 2, involving a depth estimation task under varied surface complexities and illuminations. In addition, we explored the effects of creating a virtual "hole" in the surface. Subjects' depth estimation accuracy and confidence were a ssessed. Results showed more depth estimation variation with HoloLens and significant depth error beneath complex occluding surfaces. However, creating a virtual hole significantly reduced depth errors and increased subjects' confidence, irrespective of accuracy enhancement. These findings have important implications for the design and use of mixed-reality technologies in surgical applications, and industrial applications such as using virtual content to guide maintenance or repair of components hidden beneath the opaque outer surface of equipment. A free copy of this paper and all supplemental materials are available at https://bit.ly/3YbkwjU.

Occlusion Handling in Augmented Reality: Past, Present and Future

One of the main goals of many augmented reality applications is to provide a seamless integration of a real scene with additional virtual data. To fully achieve that goal, such applications must typically provide high-quality real-world tracking, support real-time performance and handle the mutual occlusion problem, estimating the position of the virtual data into the real scene and rendering the virtual content accordingly. In this survey, we focus on the occlusion handling problem in augmented reality applications and provide a detailed review of 161 articles published in this field between January 1992 and August 2020. To do so, we present a historical overview of the most common strategies employed to determine the depth order between real and virtual objects, to visualize hidden objects in a real scene, and to build occlusion-capable visual displays. Moreover, we look at the state-of-the-art techniques, highlight the recent research trends, discuss the current open problems of occlusion handling in augmented reality, and suggest future directions for research.

Medical Augmented Reality: Definition, Principle Components, Domain Modeling, and Design-Development-Validation Process

Three decades after the first set of work on Medical Augmented Reality (MAR) was presented to the international community, and ten years after the deployment of the first MAR solutions into operating rooms, its exact definition, basic components, systematic design, and validation still lack a detailed discussion. This paper defines the basic components of any Augmented Reality (AR) solution and extends them to exemplary Medical Augmented Reality Systems (MARS). We use some of the original MARS applications developed at the Chair for Computer Aided Medical Procedures and deployed into medical schools for teaching anatomy and into operating rooms for telemedicine and surgical guidance throughout the last decades to identify the corresponding basic components. In this regard, the paper is not discussing all past or existing solutions but only aims at defining the principle components and discussing the particular domain modeling for MAR and its design-development-validation process, and providing exemplary cases through the past in-house developments of such solutions.

The Impact of Focus and Context Visualization Techniques on Depth Perception in Optical See-Through Head-Mounted Displays

Estimating the depth of virtual content has proven to be a challenging task in Augmented Reality (AR) applications. Existing studies have shown that the visual system makes use of multiple depth cues to infer the distance of objects, occlusion being one of the most important ones. The ability to generate appropriate occlusions becomes particularly important for AR applications that require the visualization of augmented objects placed below a real surface. Examples of these applications are medical scenarios in which the visualization of anatomical information needs to be observed within the patient's body. In this regard, existing works have proposed several focus and context (F+C) approaches to aid users in visualizing this content using Video See-Through (VST) Head-Mounted Displays (HMDs). However, the implementation of these approaches in Optical See-Through (OST) HMDs remains an open question due to the additive characteristics of the display technology. In this article, we, for the first time, design and conduct a user study that compares depth estimation between VST and OST HMDs using existing in-situ visualization methods. Our results show that these visualizations cannot be directly transferred to OST displays without increasing error in depth perception tasks. To tackle this gap, we perform a structured decomposition of the visual properties of AR F+C methods to find best-performing combinations. We propose the use of chromatic shadows and hatching approaches transferred from computer graphics. In a second study, we perform a factorized analysis of these combinations, showing that varying the shading type and using colored shadows can lead to better depth estimation when using OST HMDs.

Optical design and pupil swim analysis of a compact, large EPD and immersive VR head mounted display

Virtual reality head-mounted displays (VR-HMDs) are crucial to Metaverse which appears to be one of the most popular terms to have been adopted over the internet recently. It provides basic infrastructure and entrance to cater for the next evolution of social interaction, and it has already been widely used in many fields. The VR-HMDs with traditional aspherical or Fresnel optics are not suitable for long-term usage because of the image quality, system size, and weight. In this study, we designed and developed a large exit pupil diameter (EPD), compact, and lightweight VR-HMD with catadioptric optics. The mathematical formula for designing the catadioptric VR optics is derived. The reason why this kind of immersive VR optics could achieve a compact size and large EPD simultaneously is answered. Various catadioptric forms are systematically proposed and compared. The design can achieve a diagonal field of view (FOV) of 96° at -1 diopter, with an EPD of 10 mm at 11 mm eye relief (ERF). The overall length (OAL) of the system was less than 20 mm. A prototype of a compact catadioptric VR-HMD system was successfully developed.

Perception! Immersion! Empowerment! Superpowers as Inspiration for Visualization

We explore how the lens of fictional superpowers can help characterize how visualizations empower people and provide inspiration for new visualization systems. Researchers and practitioners often tout visualizations' ability to "make the invisible visible" and to "enhance cognitive abilities." Meanwhile superhero comics and other modern fiction often depict characters with similarly fantastic abilities that allow them to see and interpret the world in ways that transcend traditional human perception. We investigate the intersection of these domains, and show how the language of superpowers can be used to characterize existing visualization systems and suggest opportunities for new and empowering ones. We introduce two frameworks: The first characterizes seven underlying mechanisms that form the basis for a variety of visual superpowers portrayed in fiction. The second identifies seven ways in which visualization tools and interfaces can instill a sense of empowerment in the people who use them. Building on these observations, we illustrate a diverse set of "visualization superpowers" and highlight opportunities for the visualization community to create new systems and interactions that empower new experiences with data Material and illustrations are available under CC-BY 4.0 at osf.io/8yhfz.

Augmented Reality for Retrosigmoid Craniotomy Planning

While medical imaging data have traditionally been viewed on two-dimensional (2D) displays, augmented reality (AR) allows physicians to project the medical imaging data on patient's bodies to locate important anatomy. We present a surgical AR application to plan the retrosigmoid craniotomy, a standard approach to access the posterior fossa and the internal auditory canal. As a simple and accurate alternative to surface landmarks and conventional surgical navigation systems, our AR application augments the surgeon's vision to guide the optimal location of cortical bone removal. In this work, two surgeons performed a retrosigmoid approach 14 times on eight cadaver heads. In each case, the surgeon manually aligned a computed tomography (CT)-derived virtual rendering of the sigmoid sinus on the real cadaveric heads using a see-through AR display, allowing the surgeon to plan and perform the craniotomy accordingly. Postprocedure CT scans were acquired to assess the accuracy of the retrosigmoid craniotomies with respect to their intended location relative to the dural sinuses. The two surgeons had a mean margin of davg = 0.6 ± 4.7 mm and davg = 3.7 ± 2.3 mm between the osteotomy border and the dural sinuses over all their cases, respectively, and only positive margins for 12 of the 14 cases. The intended surgical approach to the internal auditory canal was successfully achieved in all cases using the proposed method, and the relatively small and consistent margins suggest that our system has the potential to be a valuable tool to facilitate planning a variety of similar skull-base procedures.

Novel Multimodal, Multiscale Imaging System with Augmented Reality

A novel multimodal, multiscale imaging system with augmented reality capability were developed and characterized. The system offers 3D color reflectance imaging, 3D fluorescence imaging, and augmented reality in real time. Multiscale fluorescence imaging was enabled by developing and integrating an in vivo fiber-optic microscope. Real-time ultrasound-fluorescence multimodal imaging used optically tracked fiducial markers for registration. Tomographical data are also incorporated using optically tracked fiducial markers for registration. Furthermore, we characterized system performance and registration accuracy in a benchtop setting. The multiscale fluorescence imaging facilitated assessing the functional status of tissues, extending the minimal resolution of fluorescence imaging to ~17.5 µm. The system achieved a mean of Target Registration error of less than 2 mm for registering fluorescence images to ultrasound images and MRI-based 3D model, which is within clinically acceptable range. The low latency and high frame rate of the prototype system has shown the promise of applying the reported techniques in clinically relevant settings in the future.

[Interactive augmented reality systems : Aid for personalized patient education and rehabilitation]

BACKGROUND

During patient education, information exchange plays a critical role both for patient compliance during medical or rehabilitative treatment and for obtaining an informed consent for an operative procedure.

OBJECTIVE

In this article the augmented reality system "Magic Mirror" as an additive tool during patient education, rehabilitation as well as anatomical education is highlighted.

MATERIAL AND METHODS

The Magic Mirror system allows the user of the system to inspect both a detailed model of the 3‑dimensional anatomy of the human body and volumetric slice images in a virtual mirror environment.

RESULTS

First preliminary results from the areas of rehabilitation and learning anatomy indicate the broad potential of the Magic Mirror. Similarly, the system also provides interesting advantages for patient education situations in comparison to traditional methods of information exchange.

CONCLUSION

Novel technologies, such as augmented reality are a door opener for many innovations in medicine. In the future, patient-specific systems, such as the Magic Mirror will be used increasingly more in areas such as patient education and rehabilitation. In order to maximize the benefits of such systems, further evaluation studies are necessary to find out about the best use cases and to start an iterative optimization process of these systems.

Augmented Reality in the Health Domain: Projecting Spatial Augmented Reality Visualizations on a Perceiver's Body for Health Communication Effects

An experiment is reported that studied the effects of spatial embodiment in augmented reality on medical attitudes about the self. College students (N = 90) viewed public service announcements (PSAs) with overlaid virtual fetuses and X-rayed images of lungs on various interfaces representing embodiment-a two-dimensional screen, a three-dimensional (3D) mannequin, and the participants' bodies (3D). Results indicated that PSA messages with richer embodied interfaces increase the sense of "being there," also known as spatial presence (SP), in sequential order; this leads to increased negative emotion regarding smoking cigarettes and an increased willingness to engage with a cigarette cessation campaign. When the SP mediates the dual model process, only affective attitudes increase the behavioral intention to engage with the campaign.

Image-Aligned Dynamic Liver Reconstruction Using Intra-Operative Field of Views for Minimal Invasive Surgery

During hepatic minimal invasive surgery (MIS), 3-D reconstruction of a liver surface by interpreting the geometry of its soft tissues is achieving attractions. One of the major issues to be addressed in MIS is liver deformation. Moreover, it severely inhibits free sight and dexterity of tissue manipulation, which causes its intra-operative morphology and soft tissue motion altered as compared to its pre-operative shape. While many applications focus on 3-D reconstruction of rigid or semi-rigid scenes, the techniques applied in hepatic MIS must be able to cope with a dynamic and deformable environment. We propose an efficient technique for liver surface reconstruction based on the structure from motion to handle liver deformation. The reconstructed liver will assist surgeons to visualize liver surface more efficiently with better depth perception. We use the intra-operative field of views to generate 3-D template mesh from a dense keypoint cloud. We estimate liver deformation by finding best correspondence between 3-D templates and reconstruct a liver image to calculate translation and rotational motions. Our technique then finely tunes deformed surface by adding smoothness using shading cues. Up till now, this technique is not used for solving the human liver deformation problem. Our approach is tested and validated with synthetic as well as real in vivo data, which reveal that the reconstruction accuracy can be enhanced using our approach even in challenging laparoscopic environments.

Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display

PURPOSE

Optical see-through head-mounted displays (OST-HMD) feature an unhindered and instantaneous view of the surgery site and can enable a mixed reality experience for surgeons during procedures. In this paper, we present a systematic approach to identify the criteria for evaluation of OST-HMD technologies for specific clinical scenarios, which benefit from using an object-anchored 2D-display visualizing medical information.

METHODS

Criteria for evaluating the performance of OST-HMDs for visualization of medical information and its usage are identified and proposed. These include text readability, contrast perception, task load, frame rate, and system lag. We choose to compare three commercially available OST-HMDs, which are representatives of currently available head-mounted display technologies. A multi-user study and an offline experiment are conducted to evaluate their performance.

RESULTS

Statistical analysis demonstrates that Microsoft HoloLens performs best among the three tested OST-HMDs, in terms of contrast perception, task load, and frame rate, while ODG R-7 offers similar text readability. The integration of indoor localization and fiducial tracking on the HoloLens provides significantly less system lag in a relatively motionless scenario.

CONCLUSIONS

With ever more OST-HMDs appearing on the market, the proposed criteria could be used in the evaluation of their suitability for mixed reality surgical intervention. Currently, Microsoft HoloLens may be more suitable than ODG R-7 and Epson Moverio BT-200 for clinical usability in terms of the evaluated criteria. To the best of our knowledge, this is the first paper that presents a methodology and conducts experiments to evaluate and compare OST-HMDs for their use as object-anchored 2D-display during interventions.

Depth Perception of Surgeons in Minimally Invasive Surgery

Minimally invasive surgery (MIS) poses visual challenges to the surgeons. In MIS, binocular disparity is not freely available for surgeons, who are required to mentally rebuild the 3-dimensional (3D) patient anatomy from a limited number of monoscopic visual cues. The insufficient depth cues from the MIS environment could cause surgeons to misjudge spatial depth, which could lead to performance errors thus jeopardizing patient safety. In this article, we will first discuss the natural human depth perception by exploring the main depth cues available for surgeons in open procedures. Subsequently, we will reveal what depth cues are lost in MIS and how surgeons compensate for the incomplete depth presentation. Next, we will further expand our knowledge by exploring some of the available solutions for improving depth presentation to surgeons. Here we will review the innovative approaches (multiple 2D camera assembly, shadow introduction) and devices (3D monitors, head-mounted devices, and auto-stereoscopic monitors) for 3D image presentation from the past few years.

Using smart glasses for ultrasound diagnostics

Ultrasound has been established as a diagnostic tool in a wide range of applications. Especially for beginners, the alignment of sectional images to patient’s spatial anatomy can be cumbersome. A direct view onto the patient’s anatomy while regarding ultrasound images may help to overcome unergonomic examination.

To solve these issues an affordable augmented reality system using smart glasses was created, that displays a (virtual) ultrasound image beneath a (real) ultrasound transducer.

Rapid prototyping-assisted maxillofacial reconstruction

Rapid prototyping (RP) technologies have found many uses in dentistry, and especially oral and maxillofacial surgery, due to its ability to promote product development while at the same time reducing cost and depositing a part of any degree of complexity theoretically. This paper provides an overview of RP technologies for maxillofacial reconstruction covering both fundamentals and applications of the technologies. Key fundamentals of RP technologies involving the history, characteristics, and principles are reviewed. A number of RP applications to the main fields of oral and maxillofacial surgery, including restoration of maxillofacial deformities and defects, reduction of functional bone tissues, correction of dento-maxillofacial deformities, and fabrication of maxillofacial prostheses, are discussed. The most remarkable challenges for development of RP-assisted maxillofacial surgery and promising solutions are also elaborated.

Training for planning tumour resection: augmented reality and human factors

Planning surgical interventions is a complex task, demanding a high degree of perceptual, cognitive, and sensorimotor skills to reduce intra- and post-operative complications. This process requires spatial reasoning to coordinate between the preoperatively acquired medical images and patient reference frames. In the case of neurosurgical interventions, traditional approaches to planning tend to focus on providing a means for visualizing medical images, but rarely support transformation between different spatial reference frames. Thus, surgeons often rely on their previous experience and intuition as their sole guide is to perform mental transformation. In case of junior residents, this may lead to longer operation times or increased chance of error under additional cognitive demands. In this paper, we introduce a mixed augmented-/virtual-reality system to facilitate training for planning a common neurosurgical procedure, brain tumour resection. The proposed system is designed and evaluated with human factors explicitly in mind, alleviating the difficulty of mental transformation. Our results indicate that, compared to conventional planning environments, the proposed system greatly improves the nonclinicians' performance, independent of the sensorimotor tasks performed ( ). Furthermore, the use of the proposed system by clinicians resulted in a significant reduction in time to perform clinically relevant tasks ( ). These results demonstrate the role of mixed-reality systems in assisting residents to develop necessary spatial reasoning skills needed for planning brain tumour resection, improving patient outcomes.

Augmented reality in healthcare education: an integrative review

Background. The effective development of healthcare competencies poses great educational challenges. A possible approach to provide learning opportunities is the use of augmented reality (AR) where virtual learning experiences can be embedded in a real physical context. The aim of this study was to provide a comprehensive overview of the current state of the art in terms of user acceptance, the AR applications developed and the effect of AR on the development of competencies in healthcare.

Methods. We conducted an integrative review. Integrative reviews are the broadest type of research review methods allowing for the inclusion of various research designs to more fully understand a phenomenon of concern. Our review included multi-disciplinary research publications in English reported until 2012.

Results. 2529 research papers were found from ERIC, CINAHL, Medline, PubMed, Web of Science and Springer-link. Three qualitative, 20 quantitative and 2 mixed studies were included. Using a thematic analysis, we’ve described three aspects related to the research, technology and education. This study showed that AR was applied in a wide range of topics in healthcare education. Furthermore acceptance for AR as a learning technology was reported among the learners and its potential for improving different types of competencies.

Discussion. AR is still considered as a novelty in the literature. Most of the studies reported early prototypes. Also the designed AR applications lacked an explicit pedagogical theoretical framework. Finally the learning strategies adopted were of the traditional style ‘see one, do one and teach one’ and do not integrate clinical competencies to ensure patients’ safety.

On mixed reality environments for minimally invasive therapy guidance: systems architecture, successes and challenges in their implementation from laboratory to clinic

Mixed reality environments for medical applications have been explored and developed over the past three decades in an effort to enhance the clinician's view of anatomy and facilitate the performance of minimally invasive procedures. These environments must faithfully represent the real surgical field and require seamless integration of pre- and intra-operative imaging, surgical instrument tracking, and display technology into a common framework centered around and registered to the patient. However, in spite of their reported benefits, few mixed reality environments have been successfully translated into clinical use. Several challenges that contribute to the difficulty in integrating such environments into clinical practice are presented here and discussed in terms of both technical and clinical limitations. This article should raise awareness among both developers and end-users toward facilitating a greater application of such environments in the surgical practice of the future.

Advanced 3-Dimensional Planning in Neurosurgery

During the past decades, medical applications of virtual reality technology have been developing rapidly, ranging from a research curiosity to a commercially and clinically important area of medical informatics and technology. With the aid of new technologies, the user is able to process large amounts of data sets to create accurate and almost realistic reconstructions of anatomic structures and related pathologies. As a result, a 3-diensional (3-D) representation is obtained, and surgeons can explore the brain for planning or training. Further improvement such as a feedback system increases the interaction between users and models by creating a virtual environment. Its use for advanced 3-D planning in neurosurgery is described. Different systems of medical image volume rendering have been used and analyzed for advanced 3-D planning: 1 is a commercial “ready-to-go” system (Dextroscope, Bracco, Volume Interaction, Singapore), whereas the others are open-source-based software (3-D Slicer, FSL, and FreesSurfer). Different neurosurgeons at our institution experienced how advanced 3-D planning before surgery allowed them to facilitate and increase their understanding of the complex anatomic and pathological relationships of the lesion. They all agreed that the preoperative experience of virtually planning the approach was helpful during the operative procedure. Virtual reality for advanced 3-D planning in neurosurgery has achieved considerable realism as a result of the available processing power of modern computers. Although it has been found useful to facilitate the understanding of complex anatomic relationships, further effort is needed to increase the quality of the interaction between the user and the model.

Augmented reality intravenous injection simulator based 3D medical imaging for veterinary medicine

Augmented reality (AR) is a technology which enables users to see the real world, with virtual objects superimposed upon or composited with it. AR simulators have been developed and used in human medicine, but not in veterinary medicine. The aim of this study was to develop an AR intravenous (IV) injection simulator to train veterinary and pre-veterinary students to perform canine venipuncture. Computed tomographic (CT) images of a beagle dog were scanned using a 64-channel multidetector. The CT images were transformed into volumetric data sets using an image segmentation method and were converted into a stereolithography format for creating 3D models. An AR-based interface was developed for an AR simulator for IV injection. Veterinary and pre-veterinary student volunteers were randomly assigned to an AR-trained group or a control group trained using more traditional methods (n = 20/group; n = 8 pre-veterinary students and n = 12 veterinary students in each group) and their proficiency at IV injection technique in live dogs was assessed after training was completed. Students were also asked to complete a questionnaire which was administered after using the simulator. The group that was trained using an AR simulator were more proficient at IV injection technique using real dogs than the control group (P ≤ 0.01). The students agreed that they learned the IV injection technique through the AR simulator. Although the system used in this study needs to be modified before it can be adopted for veterinary educational use, AR simulation has been shown to be a very effective tool for training medical personnel. Using the technology reported here, veterinary AR simulators could be developed for future use in veterinary education.

DVV: a taxonomy for mixed reality visualization in image guided surgery

Mixed reality visualizations are increasingly studied for use in image guided surgery (IGS) systems, yet few mixed reality systems have been introduced for daily use into the operating room (OR). This may be the result of several factors: the systems are developed from a technical perspective, are rarely evaluated in the field, and/or lack consideration of the end user and the constraints of the OR. We introduce the Data, Visualization processing, View (DVV) taxonomy which defines each of the major components required to implement a mixed reality IGS system. We propose that these components be considered and used as validation criteria for introducing a mixed reality IGS system into the OR. A taxonomy of IGS visualization systems is a step toward developing a common language that will help developers and end users discuss and understand the constituents of a mixed reality visualization system, facilitating a greater presence of future systems in the OR. We evaluate the DVV taxonomy based on its goodness of fit and completeness. We demonstrate the utility of the DVV taxonomy by classifying 17 state-of-the-art research papers in the domain of mixed reality visualization IGS systems. Our classification shows that few IGS visualization systems' components have been validated and even fewer are evaluated.

A guide to stereoscopic 3D displays in medicine

Stereoscopic displays can potentially improve many aspects of medicine. However, weighing the advantages and disadvantages of such displays remains difficult, and more insight is needed to evaluate whether stereoscopic displays are worth adopting. In this article, we begin with a review of monocular and binocular depth cues. We then apply this knowledge to examine how stereoscopic displays can potentially benefit diagnostic imaging, medical training, and surgery. It is apparent that the binocular depth information afforded by stereo displays 1) aid the detection of diagnostically relevant shapes, orientations, and positions of anatomical features, especially when monocular cues are absent or unreliable; 2) help novice surgeons orient themselves in the surgical landscape and perform complicated tasks; and 3) improve the three-dimensional anatomical understanding of students with low visual-spatial skills. The drawbacks of stereo displays are also discussed, including extra eyewear, potential three-dimensional misperceptions, and the hurdle of overcoming familiarity with existing techniques. Finally, we list suggested guidelines for the optimal use of stereo displays. We provide a concise guide for medical practitioners who want to assess the potential benefits of stereo displays before adopting them.

Landmark-based augmented reality system for paranasal and transnasal endoscopic surgeries

BACKGROUND

In this paper we present a landmark-based augmented reality (AR) endoscope system for endoscopic paranasal and transnasal surgeries along with fast and automatic calibration and registration procedures for the endoscope.

METHODS

Preoperatively the surgeon selects natural landmarks or can define new landmarks in CT volume. These landmarks are overlaid, after proper registration of preoperative CT to the patient, on the endoscopic video stream. The specified name of the landmark, along with selected colour and its distance from the endoscope tip, is also augmented. The endoscope optics are calibrated and registered by fast and automatic methods. Accuracy of the system is evaluated in a metallic grid and cadaver set-up.

RESULTS

Root mean square (RMS) error of the system is 0.8 mm in a controlled laboratory set-up (metallic grid) and was 2.25 mm during cadaver studies.

CONCLUSIONS

A novel landmark-based AR endoscope system is implemented and its accuracy is evaluated. Augmented landmarks will help the surgeon to orientate and navigate the surgical field. Studies prove the capability of the system for the proposed application. Further clinical studies are planned in near future.

Development of 3D space-sharing interface using augmented reality technology for domestic tele-echography

We propose a domestic tele-echography system linking between the patient's home and a hospital because of the demand due to increase in the number of patients by aging society and recent progress in portable echography enabled us to develop this system. In previous researches three-dimensional position of the ultrasound probe was difficult to specify because a remote doctor observe the patient through a video camera. Therefore we have developed a reproduction system of the probe position and angle using the ARToolKit and GUI interface using OpenGL. Only an USB camera and two markers for the body surface and the probe are necessary to memorize and transfer three-dimensional position of the probe. We have also designed a doctor side interface including echogram, patient scene and CG to instruct probe operation. As a result of evaluation experiments, guided position was satisfied to reproduce the echogram for diagnosis.

Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism

It has been a challenge to design an optical see-through head-mounted display (OST-HMD) that has a wide field of view (FOV) and low f-number (f/#) while maintaining a compact, lightweight, and nonintrusive form factor. In this paper, we present an OST-HMD design using a wedge-shaped freeform prism cemented with a freeform lens. The prism, consisting of three freeform surfaces (FFSs), serves as the near-eye viewing optics that magnifies the image displayed through a microdisplay, and the freeform lens is an auxiliary element attached to the prism in order to maintain a nondistorted see-through view of a real-world scene. Both the freeform prism and the lens utilize plastic materials to achieve light weight. The overall dimension of the optical system per eye is no larger than 25 mm by 22 mm by 12 mm, and the weight is 8 g. Based on a 0.61 in. microdisplay, our system demonstrates a diagonal FOV of 53.5 degrees and an f/# of 1.875, with an 8 mm exit pupil diameter and an 18.25 mm eye relief.

Cognitive and learning sciences in biomedical and health instructional design: A review with lessons for biomedical informatics education

Theoretical and methodological advances in the cognitive and learning sciences can greatly inform curriculum and instruction in biomedicine and also educational programs in biomedical informatics. It does so by addressing issues such as the processes related to comprehension of medical information, clinical problem-solving and decision-making, and the role of technology. This paper reviews these theories and methods from the cognitive and learning sciences and their role in addressing current and future needs in designing curricula, largely using illustrative examples drawn from medical education. The lessons of this past work are also applicable, however, to biomedical and health professional curricula in general, and to biomedical informatics training, in particular. We summarize empirical studies conducted over two decades on the role of memory, knowledge organization and reasoning as well as studies of problem-solving and decision-making in medical areas that inform curricular design. The results of this research contribute to the design of more informed curricula based on empirical findings about how people learn and think, and more specifically, how expertise is developed. Similarly, the study of practice can also help to shape theories of human performance, technology-based learning, and scientific and professional collaboration that extend beyond the domain of medicine. Just as biomedical science has revolutionized health care practice, research in the cognitive and learning sciences provides a scientific foundation for education in biomedicine, the health professions, and biomedical informatics.

Pq-space based non-photorealistic rendering for augmented reality

The increasing use of robotic assisted minimally invasive surgery (MIS) provides an ideal environment for using Augmented Reality (AR) for performing image guided surgery. Seamless synthesis of AR depends on a number of factors relating to the way in which virtual objects appear and visually interact with a real environment. Traditional overlaid AR approaches generally suffer from a loss of depth perception. This paper presents a new AR method for robotic assisted MIS, which uses a novel pq-space based non-photorealistic rendering technique for providing see-through vision of the embedded virtual object whilst maintaining salient anatomical details of the exposed anatomical surface. Experimental results with both phantom and in vivo lung lobectomy data demonstrate the visual realism achieved for the proposed method and its accuracy in providing high fidelity AR depth perception.

Pose estimation from a two-dimensional view by use of composite correlation filters and neural networks

We present a technique to estimate the pose of a three-dimensional object from a two-dimensional view. We first compute the correlation between the unknown image and several synthetic-discriminant-function filters constructed with known views of the object. We consider both linear and nonlinear correlations. The filters are constructed in such a way that the obtained correlation values depend on the pose parameters. We show that this dependence is not perfectly linear, in particular for nonlinear correlation. Therefore we use a two-layer neural network to retrieve the pose parameters from the correlation values. We demonstrate the technique by simultaneously estimating the in-plane and out-of-plane orientations of an airplane within an 8-deg portion. We show that a nonlinear correlation is necessary to identify the object and also to estimate its pose. On the other hand, linear correlation is more accurate and more robust. A combination of linear and nonlinear correlations gives the best results.

Intrabody three-dimensional position sensor for an ultrasound endoscope

To avoid or reduce the X-ray exposure in endoscopic examinations and therapy, as an alternative to the conventional two-dimensional X-ray fluoroscopy we are developing an intrabody navigation system that can directly measure and visualize the three-dimensional (3-D) position of the tip and the trace of an ultrasound endoscope. The proposed system can identify the 3-D location and direction of the endoscope probe inserted into the body to furnish endoscopic images. A marker transducer(s) placed on the surface of the body transmits ultrasound pulses, which are visualized as a marker synchronized to the scanning of the endoscope. The position (direction and distance of the marker transducer(s) outside the body relative to the scanning probe inside the body) of the marker is detected and measured in the scanned image of the ultrasound endoscope. Further, an optical localizer locates the marker transducer(s) with six degrees of freedom. Thus, the proposed method performs inside-body 3-D localization by utilizing the inherent image reconstruction function of the ultrasound endoscope, and is able to be used with currently available commercial ultrasound image scanners. The system may be envisaged as a kind of global positioning system for intrabody navigation.

Augmented reality guidance for needle biopsies: An initial randomized, controlled trial in phantoms††A preliminary version of this paper was presented at the Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2001 conference in Utrecht. The Netherlands (Rosenthal et al., 2001)

We report the results of a randomized, controlled trial to compare the accuracy of standard ultrasound-guided needle biopsy to biopsies performed using a 3D Augmented Reality (AR) guidance system. A board-certified radiologist conducted 50 core biopsies of breast phantoms, with biopsies randomly assigned to one of the methods in blocks of five biopsies each. The raw ultrasound data from each biopsy was recorded. Another board-certified radiologist, blinded to the actual biopsy guidance mechanism, evaluated the ultrasound recordings and determined the distance of the biopsy from the ideal position. A repeated measures analysis of variance indicated that the head-mounted display method led to a statistically significantly smaller mean deviation from the desired target than did the standard display method (2.48 mm for control versus 1.62 mm for augmented reality, p<0.02). This result suggests that AR systems can offer improved accuracy over traditional biopsy guidance methods.

The use of virtual reality exposure in the treatment of anxiety disorders

One possible alternative to standard in vivo exposure may be virtual reality exposure. Virtual reality integrates real-time computer graphics, body tracking devices, visual displays, and other sensory input devices to immerse a participant in a computer-generated virtual environment. Virtual reality exposure (VRE) is potentially an efficient and cost-effective treatment of anxiety disorders. VRE therapy has been successful in reducing the fear of heights in the first known controlled study of virtual reality in the treatment of a psychological disorder. Outcome was assessed on measures of anxiety, avoidance, attitudes, and distress. Significant group differences were found on all measures such that the VRE group was significantly improved at posttreatment but the control group was unchanged. The efficacy of virtual reality exposure therapy was also supported for the fear of flying in a case study. The potential for virtual reality exposure treatment for these and other disorders is explored.

Computer-vision-enabled augmented reality fundus biomicroscopy

PURPOSE

To guide treatment for macular diseases and to facilitate real-time image measurement and comparison, investigations were initiated to permit overlay of previously stored photographic and angiographic images directly onto the real-time slit-lamp biomicroscopic fundus image.

DESIGN

Experimental study in model eyes, and preliminary observations in human subjects.

METHODS

A modified, binocular video slit lamp interfaced to a personal computer and framegrabber allows for image acquisition and rendering of stored images overlaid onto the real-time slit-lamp biomicroscopic fundus image. Development proceeds with rendering on a computer monitor, while construction is completed on a miniature display interfaced directly with one of the slit-lamp oculars. Registration and tracking are performed with in-house-developed software.

MAIN OUTCOME MEASURES

Tracking speed and accuracy, ergonomic acceptability.

RESULTS

Computer-vision algorithms permit robust montaging, tracking, registration, and rendering of previously stored photographic and angiographic images onto the real-time slit-lamp fundus biomicroscopic image. In model eyes and in preliminary studies in a human eye, optimized registration permits near-video-rate image overlay with updates at 3 to 10 Hz and misregistration errors on the order of 1 to 5 pixels.

CONCLUSIONS

A prototype for ophthalmic augmented reality (image overlay) is presented. The current hardware/software implementation allows for robust performance.

Three-dimensional ultrasound imaging

The objective of this article is to provide scientists, engineers and clinicians with an up-to-date overview on the current state of development in the area of three-dimensional ultrasound (3-DUS) and to serve as a reference for individuals who wish to learn more about 3-DUS imaging. The sections will review the state of the art with respect to 3-DUS imaging, methods of data acquisition, analysis and display approaches. Clinical sections summarize patient research study results to date with discussion of applications by organ system. The basic algorithms and approaches to visualization of 3-D and 4-D ultrasound data are reviewed, including issues related to interactivity and user interfaces. The implications of recent developments for future ultrasound imaging/visualization systems are considered. Ultimately, an improved understanding of ultrasound data offered by 3-DUS may make it easier for primary care physicians to understand complex patient anatomy. Tertiary care physicians specializing in ultrasound can further enhance the quality of patient care by using high-speed networks to review volume ultrasound data at specialization centers. Access to volume data and expertise at specialization centers affords more sophisticated analysis and review, further augmenting patient diagnosis and treatment.

Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization

This paper describes augmented reality visualization for the guidance of breast-conservative cancer surgery using ultrasonic images acquired in the operating room just before surgical resection. By combining an optical three-dimensional (3-D) position sensor, the position and orientation of each ultrasonic cross section are precisely measured to reconstruct geometrically accurate 3-D tumor models from the acquired ultrasonic images. Similarly, the 3-D position and orientation of a video camera are obtained to integrate video and ultrasonic images in a geometrically accurate manner. Superimposing the 3-D tumor models onto live video images of the patient's breast enables the surgeon to perceive the exact 3-D position of the tumor, including irregular cancer invasions which cannot be perceived by touch, as if it were visible through the breast skin. Using the resultant visualization, the surgeon can determine the region for surgical resection in a more objective and accurate manner, thereby minimizing the risk of a relapse and maximizing breast conservation. The system was shown to be effective in experiments using phantom and clinical data.