DIVA: Natural Navigation Inside 3D Images Using Virtual Reality

Efficient and Accurate Semi-Automatic Neuron Tracing with Extended Reality

Neuron tracing, alternately referred to as neuron reconstruction, is the procedure for extracting the digital representation of the three-dimensional neuronal morphology from stacks of microscopic images. Achieving accurate neuron tracing is critical for profiling the neuroanatomical structure at single-cell level and analyzing the neuronal circuits and projections at whole-brain scale. However, the process often demands substantial human involvement and represents a nontrivial task. Conventional solutions towards neuron tracing often contend with challenges such as non-intuitive user interactions, suboptimal data generation throughput, and ambiguous visualization. In this paper, we introduce a novel method that leverages the power of extended reality (XR) for intuitive and progressive semi-automatic neuron tracing in real time. In our method, we have defined a set of interactors for controllable and efficient interactions for neuron tracing in an immersive environment. We have also developed a GPU-accelerated automatic tracing algorithm that can generate updated neuron reconstruction in real time. In addition, we have built a visualizer for fast and improved visual experience, particularly when working with both volumetric images and 3D objects. Our method has been successfully implemented with one virtual reality (VR) headset and one augmented reality (AR) headset with satisfying results achieved. We also conducted two user studies and proved the effectiveness of the interactors and the efficiency of our method in comparison with other approaches for neuron tracing.

Evaluation of Remote Surgical Hands-on Training in Veterinary Education Using a Hololens Mixed Reality Head-Mounted Display

Conferencing system-assisted online classes have been conducted worldwide since the COVID-19 pandemic, and the use of three-dimensional glasses may improve pre-clinical veterinary education. However, students' satisfaction with this technique rather than their ability to perform surgery using these items has not been assessed. This study could potentially assess students' satisfaction with technique/instruction rather than their ability to perform surgery using these items.This study aimed to evaluate the effectiveness of remote online hands-on training in veterinary education using 3D glasses. Sixty students enrolled at the Faculty of Veterinary Medicineat Yamaguchi University voluntarily participated and were randomly divided into a 3D glasses and tablet group, each with 30 students. Each student completed one orthopedic and one ophthalmological task. The orthopedic task was performing surgery on a limb model, whereas the ophthalmological task involved incising a cornea on an eye model. The 3D glasses group participated in the ophthalmology task, then the orthopedic task, at a separate venue from the instructor. The tablet group participated in the same tasks using a tablet. In the student questionnaire, orthopedic screw fixation showed significantly higher levels of satisfaction in the 3D glasses group than in the tablet group, indicating a preference for this method. By contrast, for ophthalmic corneal suturing, the tablet group showed a significantly higher level of satisfaction than the 3D glasses group. Our findings showed that 3D glasses have a high educational value in practical training requiring depth and angle information.

Three-dimensional virtual reality in surgical planning for breast cancer with reconstruction

Breast surgery is performed to achieve local control in patients with breast cancer. Visualization of the anatomy with a virtual reality software platform reconstructed from magnetic resonance imaging data improves surgical planning with regards to volume and localization of the tumor, lymph nodes, blood vessels, and surrounding tissue to perform oncoplastic tissue rearrangement. We report the use and advantages of virtual reality added to the magnetic resonance imaging assessment in a 36-year-old woman with breast cancer who underwent nipple sparing mastectomy with tissue expander reconstruction.

3D reconstruction of the cerebellar germinal layer reveals tunneling connections between developing granule cells

The difficulty of retrieving high-resolution, in vivo evidence of the proliferative and migratory processes occurring in neural germinal zones has limited our understanding of neurodevelopmental mechanisms. Here, we used a connectomic approach using a high-resolution, serial-sectioning scanning electron microscopy volume to investigate the laminar cytoarchitecture of the transient external granular layer (EGL) of the developing cerebellum, where granule cells coordinate a series of mitotic and migratory events. By integrating image segmentation, three-dimensional reconstruction, and deep-learning approaches, we found and characterized anatomically complex intercellular connections bridging pairs of cerebellar granule cells throughout the EGL. Connected cells were either mitotic, migratory, or transitioning between these two cell stages, displaying a chronological continuum of proliferative and migratory events never previously observed in vivo at this resolution. This unprecedented ultrastructural characterization poses intriguing hypotheses about intercellular connectivity between developing progenitors and its possible role in the development of the central nervous system.

Heterogeneities of zebrafish vasculature development studied by a high throughput light-sheet flow imaging system

Zebrafish is one of the ideal model animals to study the structural and functional heterogeneities in development. However, the lack of high throughput 3D imaging techniques has limited studies to only a few samples, despite zebrafish spawning tens of embryos at once. Here, we report a light-sheet flow imaging system (LS-FIS) based on light-sheet illumination and a continuous flow imager. LS-FIS enables whole-larva 3D imaging of tens of samples within half an hour. The high throughput 3D imaging capability of LS-FIS was demonstrated with the developmental study of the zebrafish vasculature from 3 to 9 days post-fertilization. Statistical analysis shows significant variances in trunk vessel development but less in hyaloid vessel development.

Challenges of intracellular visualization using virtual and augmented reality

Microscopy image observation is commonly performed on 2D screens, which limits human capacities to grasp volumetric, complex, and discrete biological dynamics. With the massive production of multidimensional images (3D + time, multi-channels) and derived images (e.g., restored images, segmentation maps, and object tracks), scientists need appropriate visualization and navigation methods to better apprehend the amount of information in their content. New modes of visualization have emerged, including virtual reality (VR)/augmented reality (AR) approaches which should allow more accurate analysis and exploration of large time series of volumetric images, such as those produced by the latest 3D + time fluorescence microscopy. They include integrated algorithms that allow researchers to interactively explore complex spatiotemporal objects at the scale of single cells or multicellular systems, almost in a real time manner. In practice, however, immersion of the user within 3D + time microscopy data represents both a paradigm shift in human-image interaction and an acculturation challenge, for the concerned community. To promote a broader adoption of these approaches by biologists, further dialogue is needed between the bioimaging community and the VR&AR developers.

Bioinformatics and the Metaverse: Are We Ready?

COVID-19 forced humanity to think about new ways of working globally without physically being present with other people, and eXtended Reality (XR) systems (defined as Virtual Reality, Augmented Reality and Mixed Reality) offer a potentially elegant solution. Previously seen as mainly for gaming, commercial and research institutions are investigating XR solutions to solve real world problems from training, simulation, mental health, data analysis, and studying disease progression. More recently large corporations such as Microsoft and Meta have announced they are developing the Metaverse as a new paradigm to interact with the digital world. This article will look at how visualization can leverage the Metaverse in bioinformatics research, the pros and cons of this technology, and what the future may hold.

New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing

Three-dimensional imaging is at the core of medical imaging and is becoming a standard in biological research. As a result, there is an increasing need to visualize, analyze and interact with data in a natural three-dimensional context. By combining stereoscopy and motion tracking, commercial virtual reality (VR) headsets provide a solution to this critical visualization challenge by allowing users to view volumetric image stacks in a highly intuitive fashion. While optimizing the visualization and interaction process in VR remains an active topic, one of the most pressing issue is how to utilize VR for annotation and analysis of data. Annotating data is often a required step for training machine learning algorithms. For example, enhancing the ability to annotate complex three-dimensional data in biological research as newly acquired data may come in limited quantities. Similarly, medical data annotation is often time-consuming and requires expert knowledge to identify structures of interest correctly. Moreover, simultaneous data analysis and visualization in VR is computationally demanding. Here, we introduce a new procedure to visualize, interact, annotate and analyze data by combining VR with cloud computing. VR is leveraged to provide natural interactions with volumetric representations of experimental imaging data. In parallel, cloud computing performs costly computations to accelerate the data annotation with minimal input required from the user. We demonstrate multiple proof-of-concept applications of our approach on volumetric fluorescent microscopy images of mouse neurons and tumor or organ annotations in medical images.

Towards Human in the Loop Analysis of Complex Point Clouds: Advanced Visualizations, Quantifications, and Communication Features in Virtual Reality

Multiple fields in biological and medical research produce large amounts of point cloud data with high dimensionality and complexity. In addition, a large set of experiments generate point clouds, including segmented medical data or single-molecule localization microscopy. In the latter, individual molecules are observed within their natural cellular environment. Analyzing this type of experimental data is a complex task and presents unique challenges, where providing extra physical dimensions for visualization and analysis could be beneficial. Furthermore, whether highly noisy data comes from single-molecule recordings or segmented medical data, the necessity to guide analysis with user intervention creates both an ergonomic challenge to facilitate this interaction and a computational challenge to provide fluid interactions as information is being processed. Several applications, including our software DIVA for image stack and our platform Genuage for point clouds, have leveraged Virtual Reality (VR) to visualize and interact with data in 3D. While the visualization aspects can be made compatible with different types of data, quantifications, on the other hand, are far from being standard. In addition, complex analysis can require significant computational resources, making the real-time VR experience uncomfortable. Moreover, visualization software is mainly designed to represent a set of data points but lacks flexibility in manipulating and analyzing the data. This paper introduces new libraries to enhance the interaction and human-in-the-loop analysis of point cloud data in virtual reality and integrate them into the open-source platform Genuage. We first detail a new toolbox of communication tools that enhance user experience and improve flexibility. Then, we introduce a mapping toolbox allowing the representation of physical properties in space overlaid on a 3D mesh while maintaining a point cloud dedicated shader. We introduce later a new and programmable video capture tool in VR and desktop modes for intuitive data dissemination. Finally, we highlight the protocols that allow simultaneous analysis and fluid manipulation of data with a high refresh rate. We illustrate this principle by performing real-time inference of random walk properties of recorded trajectories with a pre-trained Graph Neural Network running in Python.

Breast Magnetic Resonance Image Analysis for Surgeons Using Virtual Reality: A Comparative Study

PURPOSE

The treatment of breast cancer, the leading cause of cancer and cancer mortality among women worldwide, is mainly on the basis of surgery. In this study, we describe the use of a medical image visualization tool on the basis of virtual reality (VR), entitled DIVA, in the context of breast cancer tumor localization among surgeons. The aim of this study was to evaluate the speed and accuracy of surgeons using DIVA for medical image analysis of breast magnetic resonance image (MRI) scans relative to standard image slice-based visualization tools.

MATERIALS AND METHODS

In our study, residents and practicing surgeons used two breast MRI reading modalities: the common slice-based radiology interface and the DIVA system in its VR mode. Metrics measured were compared in relation to postoperative anatomical-pathologic reports.

RESULTS

Eighteen breast surgeons from the Institut Curie performed all the analysis presented. The MRI analysis time was significantly lower with the DIVA system than with the slice-based visualization for residents, practitioners, and subsequently the entire group (P < .001). The accuracy of determination of which breast contained the lesion significantly increased with DIVA for residents (P = .003) and practitioners (P = .04). There was little difference between the DIVA and slice-based visualization for the determination of the number of lesions. The accuracy of quadrant determination was significantly improved by DIVA for practicing surgeons (P = .01) but not significantly for residents (P = .49).

CONCLUSION

This study indicates that the VR visualization of medical images systematically improves surgeons' analysis of preoperative breast MRI scans across several different metrics irrespective of surgeon seniority.

Fast-track virtual reality for cardiac imaging in congenital heart disease

BACKGROUND AND AIM OF THE STUDY

We sought to evaluate the appropriateness of cardiac anatomy renderings by a new virtual reality (VR) technology, entitled DIVA, directly applicable to raw magnetic resonance imaging (MRI) data without intermediate segmentation steps in comparison to standard three-dimensional (3D) rendering techniques (3D PDF and 3D printing). Differences in post-processing times were also evaluated.

METHODS

We reconstructed 3D (STL, 3D-PDF, and 3D printed ones) and VR models of three patients with different types of complex congenital heart disease (CHD). We then asked a senior pediatric heart surgeon to compare and grade the results obtained.

RESULTS

All anatomical structures were well visualized in both VR and 3D PDF/printed models. Ventricular-arterial connections and their relationship with the great vessels were better visualized with the VR model (Case 2); aortic arch anatomy and details were also better visualized by the VR model (Case 3). The median post-processing time to get VR models using DIVA was 5 min in comparison to 8 h (range 8-12 h including printing time) for 3D models (PDF/printed).

CONCLUSIONS

VR directly applied to non-segmented 3D-MRI data set is a promising technique for 3D advanced modeling in CHD. It is systematically more consistent and faster when compared to standard 3D-modeling techniques.

Partial breast resection for multifocal lower quadrant breast tumour using virtual reality

Oncoplastic surgery allows an increase in the number of indications for conservative breast cancer treatments. However, uncertainty as to whether it can be performed still exists in certain situations such as with multicentric or multifocal lesions, even when the breast volume can accommodate it. With the aid of a virtual reality software, DIVA, allowing the precise visualisation of tumours and breast volumes based entirely on the patient's MRI, we report the ability to rapidly confirm and secure an indication for partial surgery of multiple lesions in a 31-year-old patient. With the described approach, the patient did not have to suffer significant disfigurement from cancerous breast surgery without compromising safety.

DIVA, a 3D virtual reality platform, improves undergraduate craniofacial trauma education

Craniofacial fractures management is challenging to teach due to the complex anatomy of the head, even when using three-dimensional CT-scan images. DIVA is a software allowing the straightforward visualization of CT-scans in a user-friendly three-dimensional virtual reality environment. Here, we assess DIVA as an educational tool for craniofacial trauma for undergraduate medical students. Three craniofacial trauma cases (jaw fracture, naso-orbital-ethmoid complex fracture and Le Fort 3 fracture) were submitted to 50 undergraduate medical students, who had to provide diagnoses and treatment plans. Each student then filled an 8-item questionnaire assessing satisfaction, potential benefit, ease of use and tolerance. Additionally, 4 postgraduate students were requested to explore these cases and to place 6 anatomical landmarks on both virtual reality renderings and usual slice-based three-dimensional CT-scan visualizations. High degrees of satisfaction (98%) without specific tolerance issues (86%) were reported. The potential benefit in a better understanding of craniofacial trauma using virtual reality was reported by almost all students (98%). Virtual reality allowed a reliable localization of key anatomical landmarks when compared with standard three-dimensional CT-scan visualization. Virtual reality interfaces such DIVA are beneficial to medical students for a better understanding of craniofacial trauma and allow a reliable rendering of craniofacial anatomy.