ChemPreview : an augmented reality-based molecular interface

chARpack: The Chemistry Augmented Reality Package

Off-loading visualization and interaction into virtual reality (VR) using head-mounted displays (HMDs) has gained considerable popularity in simulation sciences, particularly in chemical modeling. Because of its unique way of soft immersion, augmented reality (AR) HMD technology has even more potential to be integrated into the everyday workflow of computational chemists. In this work, we present our environment to explore the prospects of AR in chemistry and general molecular sciences: The chemistry in Augmented Reality package (chARpack). Besides providing an extensible framework, our software focuses on a seamless transition between a 3D stereoscopic view with true 3D interactions and the traditional desktop PC setup to provide users with the best setup for all tasks in their workflow. Using feedback from domain experts, we discuss our design requirements for this kind of hybrid working environment (AR + PC), regarding input, features, degree of immersion, and collaboration.

Interactive molecular dynamics in virtual reality for modelling materials and catalysts

Interactive molecular dynamics simulation in virtual reality (iMD-VR) is emerging as a promising technique in molecular science. Here, we demonstrate its use in a range of fifteen applications in materials science and heterogeneous catalysis. In this work, the iMD-VR package Narupa is used with the MD package, DL_POLY [1]. We show how iMD-VR can be used to: (i) investigate the mechanism of lithium fast ion conduction by directing the formation of defects showing that vacancy transport is favoured over interstitialcy mechanisms, and (ii) guide a molecule through a zeolite pore to explore diffusion within zeolites, examining in detail the motion of methyl n-hexanoate in H-ZSM-5 zeolite and identifying bottlenecks restricting diffusion. iMD-VR allows users to manipulate these systems intuitively, to drive changes in them and observe the resulting changes in structure and dynamics. We make these simulations available, as a resource for both teaching and research. All simulation files, with videos, can be found online (https://doi.org/10.5281/zenodo.8252314) and are provided as open-source material.

A virtual reality classroom to teach and explore crystal solid state structures

We present an educational application of virtual reality that we created to help students gain an in-depth understanding of the internal structure of crystals and related key concepts. Teachers can use it to give lectures to small groups (10-15) of students in a shared virtual environment, both remotely (with teacher and students in different locations) and locally (while sharing the same physical space). Lectures can be recorded, stored in an online repository, and shared with students who can either review a recorded lecture in the same virtual environment or can use the application for self-studying by exploring a large collection of available crystal structures. We validated our application with human subjects receiving positive feedback.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s11042-022-13410-0https://doi.org/10.1007/s11042-022-13410-0.

Bringing chemical structures to life with augmented reality, machine learning, and quantum chemistry

Visualizing 3D molecular structures is crucial to understanding and predicting their chemical behavior. However, static 2D hand-drawn skeletal structures remain the preferred method of chemical communication. Here, we combine cutting-edge technologies in augmented reality (AR), machine learning, and computational chemistry to develop MolAR, an open-source mobile application for visualizing molecules in AR directly from their hand-drawn chemical structures. Users can also visualize any molecule or protein directly from its name or protein data bank ID and compute chemical properties in real time via quantum chemistry cloud computing. MolAR provides an easily accessible platform for the scientific community to visualize and interact with 3D molecular structures in an immersive and engaging way.

Conceptualizing learning about proteins with a molecular viewer in high school based on the integration of two theoretical frameworks

The use of a molecular viewer to visualize proteins has become more prevalent in high schools in recent years. We relied on the foundations of two theoretical frameworks to analyze questions in two learning tasks designed for 10th- to 12th-grade biotechnology majors that make use of Jmol. The two theoretical frameworks were: (i) classification of scientific knowledge into content, procedural, and epistemic knowledge; and (ii) evaluation of the cognitive skills central to visual literacy in biochemistry. During the analysis, two sub-elements of procedural knowledge emerged from the data: (i) the visualization of molecular models, and (ii) the use of Jmol software features. Based on the theoretical frameworks and data analysis, we suggest a conceptualization of learning about proteins using a molecular viewer, where the scientific knowledge elements are integrated with the eight cognitive skills central to visual literacy in biochemistry. In addition, a model presenting a hierarchy for the knowledge elements and sub-elements is suggested. In this model, content knowledge is a basic requirement; without it, the other knowledge elements cannot be used. Moreover, the use of epistemic knowledge or Jmol software features is not possible without visualization of the molecular models, which requires content knowledge. This conceptualization is expected to facilitate the development of learning tasks, decrease the complexity of knowledge acquisition for students; it may also assist the teacher during the teaching process.

Exploring human-guided strategies for reaction network exploration: Interactive molecular dynamics in virtual reality as a tool for citizen scientists

The emerging fields of citizen science and gamification reformulate scientific problems as games or puzzles to be solved. Through engaging the wider non-scientific community, significant breakthroughs may be made by analyzing citizen-gathered data. In parallel, recent advances in virtual reality (VR) technology are increasingly being used within a scientific context and the burgeoning field of interactive molecular dynamics in VR (iMD-VR) allows users to interact with dynamical chemistry simulations in real time. Here, we demonstrate the utility of iMD-VR as a medium for gamification of chemistry research tasks. An iMD-VR "game" was designed to encourage users to explore the reactivity of a particular chemical system, and a cohort of 18 participants was recruited to playtest this game as part of a user study. The reaction game encouraged users to experiment with making chemical reactions between a propyne molecule and an OH radical, and "molecular snapshots" from each game session were then compiled and used to map out reaction pathways. The reaction network generated by users was compared to existing literature networks demonstrating that users in VR capture almost all the important reaction pathways. Further comparisons between humans and an algorithmic method for guiding molecular dynamics show that through using citizen science to explore these kinds of chemical problems, new approaches and strategies start to emerge.

Development and use of augmented reality models to teach medicinal chemistry

BACKGROUND AND PURPOSE

Students in the doctor of pharmacy curriculum have varied backgrounds in their chemical training and also their ability to make mental conversions from two-dimensional chemical representations, on lecture slides or textbook images, to three-dimensional cognitive understanding. In order to bridge the gap, augmented reality (AR) models were developed to provide an alternative learning medium for the students. AR was selected to take advantage of the ubiquitous presence of smartphones, without incurring the expense of Virtual Reality hardware.

EDUCATIONAL ACTIVITY AND SETTING

AR models were developed and introduced in the classroom in three phases. Student survey responses were used to improve the utility of the models in between phases. Active learning exercises were developed that required both individual and group interactions to complete.

FINDINGS

An optimized AR model creation workflow was developed that allowed each AR model to be created and posted in about 30 min. Depending on the phase of the study, 69% to 88% of the students found the AR models easy to use and 58% to 83% wanted to see more AR models used in future lectures. A majority (76%) of the students viewed the AR models on their smartphones.

SUMMARY

Augmented reality modules were created for use in medicinal chemistry courses in the pharmacy curriculum. Models were introduced in phases and included iterative improvements based on student feedback. The AR exercises provided active learning opportunities and were well received. The majority of students would like additional AR modules used in the course.

Combining Virtual Reality Visualization with Ensemble Molecular Dynamics to Study Complex Protein Conformational Changes

Molecular dynamics (MD) simulations are increasingly used to elucidate relationships between protein structure, dynamics, and their biological function. Currently, it is extremely challenging to perform MD simulations of large-scale structural rearrangements in proteins that occur on millisecond timescales or beyond, as this requires very significant computational resources, or the use of cumbersome "collective variable" enhanced sampling protocols. Here, we describe a framework that combines ensemble MD simulations and virtual reality visualization (eMD-VR) to enable users to interactively generate realistic descriptions of large amplitude, millisecond timescale protein conformational changes in proteins. Detailed tests demonstrate that eMD-VR substantially decreases the computational cost of folding simulations of a WW domain, without the need to define collective variables a priori. We further show that eMD-VR generated pathways can be combined with Markov state models to describe the thermodynamics and kinetics of large-scale loop motions in the enzyme cyclophilin A. Our results suggest eMD-VR is a powerful tool for exploring protein energy landscapes in bioengineering efforts.

An accessible, open-source mobile application for macromolecular visualization using augmented reality

Understanding macromolecular structures is essential for biology education. Augmented reality (AR) applications have shown promise in science, technology, engineering, and mathematics (STEM) education, but are not widely used for protein visualization. While there are some tools for AR protein visualization, none of them are accessible to the layperson who possesses neither specialized AR hardware nor the technical skill to comfortably navigate three-dimensional (3D) rendering and file conversions. Here, we describe Palantir, an open source mobile Android application easily installable on compatible devices from the Google Play Store. Palantir does not require specialized hardware, printed image, manual 3D rendering, or file format conversion. Palantir makes AR macromolecular visualization accessible to anyone with a compatible mobile device, and we hope it finds widespread application in STEM education.

ProteinVR: Web-based molecular visualization in virtual reality

Protein structure determines biological function. Accurately conceptualizing 3D protein/ligand structures is thus vital to scientific research and education. Virtual reality (VR) enables protein visualization in stereoscopic 3D, but many VR molecular-visualization programs are expensive and challenging to use; work only on specific VR headsets; rely on complicated model-preparation software; and/or require the user to install separate programs or plugins. Here we introduce ProteinVR, a web-based application that works on various VR setups and operating systems. ProteinVR displays molecular structures within 3D environments that give useful biological context and allow users to situate themselves in 3D space. Our web-based implementation is ideal for hypothesis generation and education in research and large-classroom settings. We release ProteinVR under the open-source BSD-3-Clause license. A copy of the program is available free of charge from http://durrantlab.com/protein-vr/, and a working version can be accessed at http://durrantlab.com/pvr/.

Proteins are microscopic machines that help maintain, defend, and regulate cells. Properly understanding the three-dimensional structures of these machines–as well as the small molecules that interact with them–can advance scientific fields ranging from basic molecular biology to drug discovery. Virtual reality (VR) is a powerful tool for studying protein structures. But many current systems for viewing molecules in VR, though effective, have challenging usability limitations. We have created a new web application called ProteinVR that overcomes these challenges. ProteinVR enables VR molecular visualization in users’ browsers, without requiring them to install a separate program or plugin. It runs on a broad range of desktop, laptop, and mobile devices. For users without VR headsets, ProteinVR leverages mobile-device orientation sensors or video-game-style keyboard navigation to provide an immersive experience. We release ProteinVR as open-source software and have posted a working version at http://durrantlab.com/pvr/.

Peppy: A virtual reality environment for exploring the principles of polypeptide structure

A key learning outcome for undergraduate biochemistry classes is a thorough understanding of the principles of protein structure. Traditional approaches to teaching this material, which include two-dimensional (2D) images on paper, physical molecular modeling kits, and projections of 3D structures into 2D, are unable to fully capture the dynamic 3D nature of proteins. We have built a virtual reality application, Peppy, aimed at facilitating teaching of the principles of protein secondary structure. Rather than attempt to model molecules with the same fidelity to the underlying physical chemistry as existing, research-oriented molecular modelling approaches, we took the more straightforward approach of harnessing the Unity video game physics engine. Indeed, the simplicity and limitations of our model are strengths in a teaching context, provoking questions and thus deeper understanding. Peppy allows exploration of the relative effects of hydrogen bonding (and electrostatic interactions more generally), backbone φ/ψ angles, basic chemical structure, and steric effects on a polypeptide structure in an accessible format that is novel, dynamic, and fun to use. Apart from describing the implementation and use of Peppy, we discuss the outcomes of deploying Peppy in undergraduate biochemistry courses.

Interactive molecular dynamics in virtual reality from quantum chemistry to drug binding: An open-source multi-person framework

As molecular scientists have made progress in their ability to engineer nanoscale molecular structure, we face new challenges in our ability to engineer molecular dynamics (MD) and flexibility. Dynamics at the molecular scale differs from the familiar mechanics of everyday objects because it involves a complicated, highly correlated, and three-dimensional many-body dynamical choreography which is often nonintuitive even for highly trained researchers. We recently described how interactive molecular dynamics in virtual reality (iMD-VR) can help to meet this challenge, enabling researchers to manipulate real-time MD simulations of flexible structures in 3D. In this article, we outline various efforts to extend immersive technologies to the molecular sciences, and we introduce "Narupa," a flexible, open-source, multiperson iMD-VR software framework which enables groups of researchers to simultaneously cohabit real-time simulation environments to interactively visualize and manipulate the dynamics of molecular structures with atomic-level precision. We outline several application domains where iMD-VR is facilitating research, communication, and creative approaches within the molecular sciences, including training machines to learn potential energy functions, biomolecular conformational sampling, protein-ligand binding, reaction discovery using "on-the-fly" quantum chemistry, and transport dynamics in materials. We touch on iMD-VR's various cognitive and perceptual affordances and outline how these provide research insight for molecular systems. By synergistically combining human spatial reasoning and design insight with computational automation, technologies such as iMD-VR have the potential to improve our ability to understand, engineer, and communicate microscopic dynamical behavior, offering the potential to usher in a new paradigm for engineering molecules and nano-architectures.

Study of Commodity VR for Computational Material Sciences

Recent advancements in virtual reality (VR) devices and software environments make it possible to easily incorporate this technology for many applications, including computational materials science. For studying three-dimensional (3D) structure models and related chemical information, we focused on using a commodity VR device (VIVE) and an authoring tool (Unity). To visualize 3D chemical structures, disturbances like judder due to dropped frames should be eliminated from the VR experience to improve simulations. We propose a simple evaluation method that is straightforward for the nonexpert or novice VR user. We examine the major visualization representations including ball, ball and stick, and isosurface systems. For systematic benchmark measurements, a pendulum from the VR device was used to generate periodic oscillatory motion during measurements of a time series in frames per second (fps). For VIVE with a refresh rate of 90 Hz, judder occurred when less than 90 fps. We demonstrated the system size limitations for the results of molecular dynamics simulations of phase separation of ABA block copolymers and experimental observations of filler morphologies in rubber.

Using virtual reality for drug discovery: a promising new outlet for novel leads

Introduction: Virtual reality (VR) environments are increasingly being used by researchers in various fields in addition to being increasingly integrated into various areas of human life, ranging from videogames to different industrial uses. VR can be used to create interactive and multimodal sensory stimuli and thus offers unique advantages over other computer-based approaches for scientific research and molecular-level applications. Consequently, VR is starting to be used in novel drug development, such as in drug discovery, and rational drug design. Areas covered: In this review, the authors discuss the basic development of VR technology, including the available hardware and software. The latest advances of VR technology in novel drug development are then detailed, and the VR programs that can be applied in relevant studies are highlighted. Expert opinion: VR will lead to a revolution in pharmaceutical development. However, there are still obstacles to the successful and extensive application of VR to drug development, including the demand for further improvements to the available hardware and software and the various limitations described with regard to accuracy and precision. As technology continues to improve, the barriers to the widespread adoption of VR will diminish and VR technologies will play an increasingly important role in novel drug development.

Molecular Visualization on the Holodeck

Can virtual reality be useful for visualizing and analyzing molecular structures and three-dimensional (3D) microscopy? Uses we are exploring include studies of drug binding to proteins and the effects of mutations, building accurate atomic models in electron microscopy and x-ray density maps, understanding how immune system cells move using 3D light microscopy, and teaching schoolchildren about biomolecules that are the machinery of life. Virtual reality (VR) offers immersive display with a wide field of view and head tracking for better perception of molecular architectures and uses 6-degree-of-freedom hand controllers for simple manipulation of 3D data. Conventional computer displays with trackpad, mouse and keyboard excel at two-dimensional tasks such as writing and studying research literature, uses for which VR technology is at present far inferior. Adding VR to the conventional computing environment could improve 3D capabilities if new user-interface problems can be solved. We have developed three VR applications: ChimeraX for analyzing molecular structures and electron and light microscopy data, AltPDB for collaborative discussions around atomic models, and Molecular Zoo for teaching young students characteristics of biomolecules. Investigations over three decades have produced an extensive literature evaluating the potential of VR in research and education. Consumer VR headsets are now affordable to researchers and educators, allowing direct tests of whether the technology is valuable in these areas. We survey here advantages and disadvantages of VR for molecular biology in the context of affordable and dramatically more powerful VR and graphics hardware than has been available in the past.

Interactive Molecular Graphics for Augmented Reality Using HoloLens

Immersive technologies like stereo rendering, virtual reality, or augmented reality (AR) are often used in the field of molecular visualisation. Modern, comparably lightweight and affordable AR headsets like Microsoft’s HoloLens open up new possibilities for immersive analytics in molecular visualisation. A crucial factor for a comprehensive analysis of molecular data in AR is the rendering speed. HoloLens, however, has limited hardware capabilities due to requirements like battery life, fanless cooling and weight. Consequently, insights from best practises for powerful desktop hardware may not be transferable. Therefore, we evaluate the capabilities of the HoloLens hardware for modern, GPU-enabled, high-quality rendering methods for the space-filling model commonly used in molecular visualisation. We also assess the scalability for large molecular data sets. Based on the results, we discuss ideas and possibilities for immersive molecular analytics. Besides more obvious benefits like the stereoscopic rendering offered by the device, this specifically includes natural user interfaces that use physical navigation instead of the traditional virtual one. Furthermore, we consider different scenarios for such an immersive system, ranging from educational use to collaborative scenarios.

Augmented Reality in Scientific Publications-Taking the Visualization of 3D Structures to the Next Level

The examination of three-dimensional structural models in scientific publications allows the reader to validate or invalidate conclusions drawn by the authors. However, either due to a (temporary) lack of access to proper visualization software or a lack of proficiency, this information is not necessarily available to every reader. As the digital revolution is quickly progressing, technologies have become widely available that overcome the limitations and offer to all the opportunity to appreciate models not only in 2D, but also in 3D. Additionally, mobile devices such as smartphones and tablets allow access to this information almost anywhere, at any time. Since access to such information has only recently become standard practice, we want to outline straightforward ways to incorporate 3D models in augmented reality into scientific publications, books, posters, and presentations and suggest that this should become general practice.