The Validity of a Mixed Reality-Based Automated Functional Mobility Assessment

Is the Apple Vision Pro the Ultimate Display? A First Perspective and Survey on Entering the Wonderland of Precision Medicine

At the Worldwide Developers Conference in June 2023, Apple introduced the Vision Pro. The Apple Vision Pro (AVP) is a mixed reality headset; more specifically, it is a virtual reality device with an additional video see-through capability. The video see-through capability turns the AVP into an augmented reality (AR) device. The AR feature is enabled by streaming the real world via cameras on the (virtual reality) screens in front of the user's eyes. This is, of course, not unique and is similar to other devices, such as the Varjo XR-3 (Varjo Technologies Oy). Nevertheless, the AVP has some interesting features, such as an inside-out screen that can show the headset wearer's eyes to "outsiders," and a button on the top, called the "digital crown," that allows a seamless blend of digital content with the user's physical space by turning it. In addition, it is untethered, except for the cable to the battery, which makes the headset more agile, compared to the Varjo XR-3. This could actually come closer to "The Ultimate Display," which Ivan Sutherland had already sketched in 1965. After a great response from the media and social networks to the release, we were able to test and review the new AVP ourselves in March 2024. Including an expert survey with 13 of our colleagues after testing the AVP in our institute, this Viewpoint explores whether the AVP can overcome clinical challenges that AR especially still faces in the medical domain; we also go beyond this and discuss whether the AVP could support clinicians in essential tasks to allow them to spend more time with their patients.

Mobility, functionality and functional mobility: A review and application for canine veterinary patients

Mobility is an essential aspect of a dog's daily life. It is defined as the ability to move freely and easily and deviations from an animals' normal mobility capabilities are often an indicator of disease, injury or pain. When a dog's mobility is compromised, often functionality (ability to perform activities of daily living [ADL]), is also impeded, which can diminish an animal's quality of life. Given this, it is necessary to understand the extent to which conditions impact a dog's physiological ability to move around their environment to carry out ADL, a concept termed functional mobility. In contrast to human medicine, validated measures of canine functional mobility are currently limited. The aim of this review is to summarise the extent to which canine mobility and functionality are associated with various diseases and how mobility and functional mobility are currently assessed within veterinary medicine. Future work should focus on developing a standardised method of assessing functional mobility in dogs, which can contextualise how a wide range of conditions impact a dog's daily life. However, for a true functional mobility assessment to be developed, a greater understanding of what activities dogs do on a daily basis and movements underpinning these activities must first be established.

The Microsoft HoloLens 2 Provides Accurate Biomechanical Measures of Performance During Military-Relevant Activities in Healthy Adults

INTRODUCTION

Augmented reality systems, like the HoloLens 2 (HL2), have the potential to provide accurate assessments of mild traumatic brain injury (mTBI) symptoms in military personnel by simulating complex military scenarios while objectively measuring the user's movements with embedded motion sensors. The aim of this project was to determine if biomechanical measures of marching and squatting, derived from the HL2 motion sensors, were statistically equivalent, within 5%, to metrics derived from the gold-standard three-dimensional motion capture (MoCap) system.

MATERIALS AND METHODS

Sixty-four adults (18-45 years; 34 males) completed a squatting and a marching task under single- (motor) and dual-task (motor + cognitive) conditions. Positional data from the HL2 and MoCap were simultaneously collected during the tasks and used to calculate and compare biomechanical outcomes. The HL2's augmented reality capabilities were utilized to deliver the cognitive component of the marching dual task.

RESULTS

Equivalence testing indicated the HL2 and MoCap measures were within 5% in all squatting metrics-trial time, squat duration, squat velocity, squat depth, and dwell time. Marching metrics, including trial time, step count, stepping rate, and step interval, were also equivalent between the two systems. The mean reaction time for responses during the Stroop test was 810 (125) milliseconds per response.

CONCLUSIONS

Biomechanical outcomes characterizing performance during two common military movements, squatting and marching, were equivalent between the HL2 and MoCap systems in healthy adults. Squatting and marching are two military-relevant tasks that require strength, motor coordination, and balance to perform, all of which are known to be affected by mTBI. Taken together, the data provide support for using the HL2 platform to deliver military-specific assessment scenarios and accurately measure performance during these activities. Utilizing objective and quantitative measures of motor function may enhance the management of military mTBI and reduce unnecessary risk to service members.

Development of the Troop Readiness Evaluation With Augmented Reality Return-to-Duty (Troop READY) Platform to Aid in the Detection and Treatment of Military Mild Traumatic Brain Injury

INTRODUCTION

Mild traumatic brain injury (mTBI) is prevalent in service members (SMs); however, there is a lack of consensus on the appropriate approach to return to duty (RTD). Head-mounted augmented reality technology, such as the HoloLens 2, can create immersive, salient environments to more effectively evaluate relevant military task performance. The Troop Readiness Evaluation with Augmented Reality Return-to-Duty (READY) platform was developed to objectively quantify cognitive and motor performance during military-specific activities to create a comprehensive approach to aid in mTBI detection and facilitate appropriate RTD. The aim of this project was to detail the technical development of the Troop READY platform, the outcomes, and its potential role in the aiding detection and RTD decision-making post mTBI. The secondary aim included evaluating the safety, feasibility, and SM usability of the Troop READY platform.

MATERIALS AND METHODS

The Troop READY platform comprises three assessment modules of progressing complexity: (1) Static and Dynamic Mobility, (2) Rifle Qualification Test, and (3) Small Unit Operations Capacity-Room Breach/Clearing Exercise. The modules were completed by 137 active duty SMs. Safety was assessed through monitoring of adverse events. Feasibility was assessed using the self-directed module completion rate. Usability was measured using the Systems Usability Scale.

RESULTS

No adverse events occurred. Completion rates of the three modules ranged from 98 to 100%. In terms of usability, the mean Systems Usability Scale score of all participants was 83.92 (13.95), placing the Troop READY platform in the good-to-excellent category. Objective motor and cognitive outcomes were generated for each module.

CONCLUSION

The Troop READY platform delivers self-directed, salient assessment modules to quantify single-task, dual-task, and unit-based performance in SMs. The resultant data provide insight into SM performance through objective outcomes and identify specific areas of executive or motor function that may be slow to recover following mTBI.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Digitizing a Therapeutic: Development of an Augmented Reality Dual-Task Training Platform for Parkinson's Disease

Augmented reality (AR) may be a useful tool for the delivery of dual-task training. This manuscript details the development of the Dual-task Augmented Reality Treatment (DART) platform for individuals with Parkinson's disease (PD) and reports initial feasibility, usability, and efficacy of the DART platform in provoking dual-task interference in individuals with PD. The DART platform utilizes the head-mounted Microsoft HoloLens2 AR device to deliver concurrent motor and cognitive tasks. Biomechanical metrics of gait and cognitive responses are automatically computed and provided to the supervising clinician. To assess feasibility, individuals with PD (N = 48) completed a bout of single-task and dual-task walking using the DART platform. Usability was assessed by the System Usability Scale (SUS). Dual-task interference was assessed by comparing single-task walking and walking during an obstacle course while performing a cognitive task. Average gait velocity decreased from 1.06 to 0.82 m/s from single- to dual-task conditions. Mean SUS scores were 81.3 (11.3), which placed the DART in the "good" to "excellent" category. To our knowledge, the DART platform is the first to use a head-mounted AR system to deliver a dual-task paradigm and simultaneously provide biomechanical data that characterize cognitive and motor performance. Individuals with PD were able to successfully use the DART platform with satisfaction, and dual-task interference was provoked. The DART platform should be investigated as a platform to treat dual-task declines associated with PD.

The Microsoft HoloLens 2 Provides Accurate Measures of Gait, Turning, and Functional Mobility in Healthy Adults

Augmented-reality (AR) headsets, such as the Microsoft HoloLens 2 (HL2), have the potential to be the next generation of wearable technology as they provide interactive digital stimuli in the context of ecologically-valid daily activities while containing inertial measurement units (IMUs) to objectively quantify the movements of the user. A necessary precursor to the widespread utilization of the HL2 in the fields of movement science and rehabilitation is the rigorous validation of its capacity to generate biomechanical outcomes comparable to gold standard outcomes. This project sought to determine equivalency of kinematic outcomes characterizing lower-extremity function derived from the HL2 and three-dimensional (3D) motion capture systems (MoCap). Sixty-six healthy adults completed two lower-extremity tasks while kinematic data were collected from the HL2 and MoCap: (1) continuous walking and (2) timed up-and-go (TUG). For all the continuous walking metrics (cumulative distance, time, number of steps, step and stride length, and velocity), equivalence testing indicated that the HL2 and MoCap were statistically equivalent (error ≤ 5%). The TUG metrics, including turn duration and turn velocity, were also statistically equivalent between the two systems. The accurate quantification of gait and turning using a wearable such as the HL2 provides initial evidence for its use as a platform for the development and delivery of gait and mobility assessments, including the in-person and remote delivery of highly salient digital movement assessments and rehabilitation protocols.

Validity of Hololens Augmented Reality Head Mounted Display for Measuring Gait Parameters in Healthy Adults and Children with Cerebral Palsy

Serious games are a promising approach to improve gait rehabilitation for people with gait disorders. Combined with wearable augmented reality headset, serious games for gait rehabilitation in a clinical setting can be envisaged, allowing to evolve in a real environment and provide fun and feedback to enhance patient’s motivation. This requires a method to obtain accurate information on the spatiotemporal gait parameters of the playing patient. To this end, we propose a new algorithm called HoloStep that computes spatiotemporal gait parameters using only the head pose provided by an augmented reality headset (Hololens). It is based on the detection of peaks associated to initial contact event, and uses a combination of locking distance, locking time, peak amplitude detection with custom thresholds for children with CP. The performance of HoloStep was compared during a walking session at comfortable speed to Zeni’s reference algorithm, which is based on kinematics and a full 3D motion capture system. Our study included 62 children with cerebral palsy (CP), classified according to Gross Motor Function Classification System (GMFCS) between levels I and III, and 13 healthy participants (HP). Metrics such as sensitivity, specificity, accuracy and precision for step detection with HoloStep were above 96%. The Intra-Class Coefficient between steps length calculated with HoloStep and the reference was 0.92 (GMFCS I), 0.86 (GMFCS II/III) and 0.78 (HP). HoloStep demonstrated good performance when applied to a wide range of gait patterns, including children with CP using walking aids. Findings provide important insights for future gait intervention using augmented reality games for children with CP.

Quantifying Spatiotemporal Gait Parameters with HoloLens in Healthy Adults and People with Parkinson's Disease: Test-Retest Reliability, Concurrent Validity, and Face Validity

Microsoft’s HoloLens, a mixed-reality headset, provides, besides holograms, rich position data of the head, which can be used to quantify what the wearer is doing (e.g., walking) and to parameterize such acts (e.g., speed). The aim of the current study is to determine test-retest reliability, concurrent validity, and face validity of HoloLens 1 for quantifying spatiotemporal gait parameters. This was done in a group of 23 healthy young adults (mean age 21 years) walking at slow, comfortable, and fast speeds, as well as in a group of 24 people with Parkinson’s disease (mean age 67 years) walking at comfortable speed. Walking was concurrently measured with HoloLens 1 and a previously validated markerless reference motion-registration system. We comprehensively evaluated HoloLens 1 for parameterizing walking (i.e., walking speed, step length and cadence) in terms of test-retest reliability (i.e., consistency over repetitions) and concurrent validity (i.e., between-systems agreement), using the intraclass correlation coefficient (ICC) and Bland–Altman’s bias and limits of agreement. Test-retest reliability and between-systems agreement were excellent for walking speed (ICC ≥ 0.861), step length (ICC ≥ 0.884), and cadence (ICC ≥ 0.765), with narrower between-systems than over-repetitions limits of agreement. Face validity was demonstrated with significantly different walking speeds, step lengths and cadences over walking-speed conditions. To conclude, walking speed, step length, and cadence can be reliably and validly quantified from the position data of the wearable HoloLens 1 measurement system, not only for a broad range of speeds in healthy young adults, but also for self-selected comfortable speed in people with Parkinson’s disease.

Avoiding 3D Obstacles in Mixed Reality: Does It Differ from Negotiating Real Obstacles?

Mixed-reality technologies are evolving rapidly, allowing for gradually more realistic interaction with digital content while moving freely in real-world environments. In this study, we examined the suitability of the Microsoft HoloLens mixed-reality headset for creating locomotor interactions in real-world environments enriched with 3D holographic obstacles. In Experiment 1, we compared the obstacle-avoidance maneuvers of 12 participants stepping over either real or holographic obstacles of different heights and depths. Participants' avoidance maneuvers were recorded with three spatially and temporally integrated Kinect v2 sensors. Similar to real obstacles, holographic obstacles elicited obstacle-avoidance maneuvers that scaled with obstacle dimensions. However, with holographic obstacles, some participants showed dissimilar trail or lead foot obstacle-avoidance maneuvers compared to real obstacles: they either consistently failed to raise their trail foot or crossed the obstacle with extreme lead-foot margins. In Experiment 2, we examined the efficacy of mixed-reality video feedback in altering such dissimilar avoidance maneuvers. Participants quickly adjusted their trail-foot crossing height and gradually lowered extreme lead-foot crossing heights in the course of mixed-reality video feedback trials, and these improvements were largely retained in subsequent trials without feedback. Participant-specific differences in real and holographic obstacle avoidance notwithstanding, the present results suggest that 3D holographic obstacles supplemented with mixed-reality video feedback may be used for studying and perhaps also training 3D obstacle avoidance.