Multisensory Integration as per Technological Advances: A Review

Multisensory integration research has allowed us to better understand how humans integrate sensory information to produce a unitary experience of the external world. However, this field is often challenged by the limited ability to deliver and control sensory stimuli, especially when going beyond audio-visual events and outside laboratory settings. In this review, we examine the scope and challenges of new technology in the study of multisensory integration in a world that is increasingly characterized as a fusion of physical and digital/virtual events. We discuss multisensory integration research through the lens of novel multisensory technologies and, thus, bring research in human-computer interaction, experimental psychology, and neuroscience closer together. Today, for instance, displays have become volumetric so that visual content is no longer limited to 2D screens, new haptic devices enable tactile stimulation without physical contact, olfactory interfaces provide users with smells precisely synchronized with events in virtual environments, and novel gustatory interfaces enable taste perception through levitating stimuli. These technological advances offer new ways to control and deliver sensory stimulation for multisensory integration research beyond traditional laboratory settings and open up new experimentations in naturally occurring events in everyday life experiences. Our review then summarizes these multisensory technologies and discusses initial insights to introduce a bridge between the disciplines in order to advance the study of multisensory integration.

How to Observe Users' Movements in Virtual Environments: Viewpoint Control in a Power Wheelchair Simulator

OBJECTIVE

We describe a networked, two-user virtual reality (VR) power wheelchair (PWC) simulator system in which an actor (client) and an observer (clinician) meet. We then present a study with 15 observers (expert clinicians) evaluating the effect of three principal forms of viewpoint control (egocentric-egomotion, egocentric-tethered, and client-centric) on the observer's assessment of driving tasks in a virtual environment (VE).

BACKGROUND

VR allows for the simulation and assessment of real-world tasks in a controlled, safe, and repeatable environment. Observing users' movement behavior in such a VE requires appropriate viewpoint control for the observer. The VR viewpoint user interface should allow an observer to make judgments equivalent or even superior to real-world situations.

METHOD

A purpose-built VR PWC simulator was developed. In a series of PWC driving tasks, we measured the perceived ease of use and sense of presence of the observers and compared the virtual assessment with real-world "gold standard" scores, including confidence levels in judgments.

RESULTS

Findings suggest that with more immersive techniques, such as egomotion and tethered egocentric viewpoints, judgments are both more accurate and more confident. The ability to walk and/or orbit around the view significantly affected the observers' sense of presence.

CONCLUSION

Incorporating the observer into the VE, through egomotion, is an effective method for assessing users' behavior in VR with implications for the transferability of virtual experiences to the real world.

APPLICATION

Our application domain serves as a representative example for tasks where the movement of users through a VE needs to be evaluated.

User Centered Design of Interaction Techniques for VR-Based Automotive Design Reviews

The exterior design is one of the most important selling propositions in the automotive premium market. Because of progressing digitization in the automotive industry, it is increasingly assessed using virtual 3D models. In this context, Virtual Reality (VR) is a key technology of continuously growing importance. However, complicated interaction in VR proves to be a major drawback in industrial settings. In this paper, we report insights of our user centered approach aiming at appropriate VR interaction techniques supporting designers, engineers, and management executives optimally in design assessment. Our approach splits into two iterations according to the main interaction tasks Visual Inspection and Model Comparison. In each iteration, alternative interaction techniques were conceptualized, implemented as prototypes, and evaluated in a user study in terms of Usability, User Experience, Intuitiveness and Task Load. In the first iteration six interaction techniques for Visual Inspection, two speech-based, two gesture-based, and two touch-based variants, were studied. Incorporating the results, the second iteration explores three interaction techniques for Model Comparison utilizing (1) a portable touch remote, (2) hand and body gestures, and (3) a multimodal mix of both. The final concepts yielded high ratings by the participants, but showed significant differences between rational and emotional aspects. We conclude that the acceptance of VR in automotive design could be facilitated by refining and applying these interaction techniques.

Using the PhysX engine for physics-based virtual surgery with force feedback

BACKGROUND

The development of modern surgical simulators is highly challenging, as they must support complex simulation environments. The demand for higher realism in such simulators has driven researchers to adopt physics-based models, which are computationally very demanding. This poses a major problem, since real-time interactions must permit graphical updates of 30 Hz and a much higher rate of 1 kHz for force feedback (haptics). Recently several physics engines have been developed which offer multi-physics simulation capabilities, including rigid and deformable bodies, cloth and fluids. While such physics engines provide unique opportunities for the development of surgical simulators, their higher latencies, compared to what is necessary for real-time graphics and haptics, offer significant barriers to their use in interactive simulation environments.

METHODS

In this work, we propose solutions to this problem and demonstrate how a multimodal surgical simulation environment may be developed based on NVIDIA's PhysX physics library. Hence, models that are undergoing relatively low-frequency updates in PhysX can exist in an environment that demands much higher frequency updates for haptics. We use a collision handling layer to interface between the physical response provided by PhysX and the haptic rendering device to provide both real-time tissue response and force feedback.

RESULTS

Our simulator integrates a bimanual haptic interface for force feedback and per-pixel shaders for graphics realism in real time. To demonstrate the effectiveness of our approach, we present the simulation of the laparoscopic adjustable gastric banding (LAGB) procedure as a case study.

CONCLUSIONS

To develop complex and realistic surgical trainers with realistic organ geometries and tissue properties demands stable physics-based deformation methods, which are not always compatible with the interaction level required for such trainers. We have shown that combining different modelling strategies for behaviour, collision and graphics is possible and desirable. Such multimodal environments enable suitable rates to simulate the major steps of the LAGB procedure.

activeNotes: Computer-Assisted Creation of Patient Progress Notes

We present activeNotes, a prototype application that supports the creation of Critical Care Notes by physicians in a hospital intensive care unit. activeNotes integrates automated, context-sensitive patient data retrieval and user control of automated data updates and alerts into the note-creation process. In a user study at New York Presbyterian Hospital, we gathered qualitative feedback on the prototype from 15 physicians. The physicians found activeNotes to be valuable and said they would use it to create both formal notes for medical records and informal notes. One surprising finding is that while physicians have rejected template-based clinical documentation systems in the past, they expressed a desire to use activeNotes to create personalized, physician-specific note templates to be reused with a given patient, or for a given condition.

Development of the VBLaST: a virtual basic laparoscopic skill trainer

BACKGROUND

The FLS training tool box has now been adopted by the Society of Gastrointestinal Endoscopic Surgeons (SAGES) as an official training tool for minimally invasive procedures.

METHODS

To overcome the limitations of the physical FLS training tool box, we have developed a Virtual Basic Laparoscopic Skill Trainer (VBLaSTTM) system, which is a 3D simulator that will allow trainees to acquire basic laparoscopic skill.

RESULTS

The outcome of this work is the development of an integrated visio-haptic workstation environment including force feedback devices and a stereo display interface whereby trainees can practice on virtual versions of the FLS. Realistic graphical rendering and high fidelity haptic interactions are achieved.

CONCLUSIONS

Surgical skill training is a long and tedious process of acquiring fine motor skills. It is expected that residents would start on trainers such as VBLaSTTM and after reaching a certain level of competence would progress to the more complex trainers for training on specific surgical procedures.

An efficient dynamic point algorithm for line-based collision detection in real time virtual environments involving haptics

In real time computer graphics, "interactivity" is limited to a display rate of 30 frames per second. However, in multimodal virtual environments involving haptic interactions, a much higher update rate of about 1 kHz is necessary to ensure continuous interactions and smooth transitions. The simplest and most efficient interaction paradigm in such environments is to represent the haptic cursor as a point. However, in many situations, such as those in the development of real time medical simulations involving the interactions of long slender surgical tools with soft deformable organs, such a paradigm is nonrealistic and at least a line-based interaction is desirable. While such paradigms exist, the main impediment to their widespread use is the associated computational complexity. In this paper, we introduce, for the first time, an efficient algorithm for computing the interaction of a line-shaped haptic cursor and polygonal surface models which has a near constant complexity. The algorithm relies on space-time coherence, topological information, and the properties of lines in 3D space to maintain proximity information between a line segment and triangle meshes. For interaction with convex objects, the line is represented by its end points and a dynamic point, which is the closest point on the line to any potentially colliding triangle. To deal with multiple contacts and non-convexities, the line is decomposed into segments and a dynamic point is used for each segment. The algorithm may be used to compute collision detection and response with rigid as well as deformable objects with no performance penalty. Realistic examples are presented to demonstrate the effectiveness of our approach.