Designing an Egocentric Video-Based Dashboard to Report Hand Performance Measures for Outpatient Rehabilitation of Cervical Spinal Cord Injury

Background

Functional use of the upper extremities (UEs) is a top recovery priority for individuals with cervical spinal cord injury (cSCI), but the inability to monitor recovery at home and limitations in hand function outcome measures impede optimal recovery.

Objectives

We developed a framework using wearable cameras to monitor hand use at home and aimed to identify the best way to report information to clinicians.

Methods

A dashboard was iteratively developed with clinician (n = 7) input through focus groups and interviews, creating low-fidelity prototypes based on recurring feedback until no new information emerged. Affinity diagramming was used to identify themes and subthemes from interview data. User stories were developed and mapped to specific features to create a high-fidelity prototype.

Results

Useful elements identified for a dashboard reporting hand performance included summaries to interpret graphs, a breakdown of hand posture and activity to provide context, video snippets to qualitatively view hand use at home, patient notes to understand patient satisfaction or struggles, and time series graphing of metrics to measure trends over time.

Conclusion

Involving end-users in the design process and breaking down user requirements into user stories helped identify necessary interface elements for reporting hand performance metrics to clinicians. Clinicians recognized the dashboard's potential to monitor rehabilitation progress, provide feedback on hand use, and track progress over time. Concerns were raised about the implementation into clinical practice, therefore further inquiry is needed to determine the tool's feasibility and usefulness in clinical practice for individuals with UE impairments.

Measuring Hand Use in the Home after Cervical Spinal Cord Injury Using Egocentric Video

Egocentric video has recently emerged as a potential solution for monitoring hand function in individuals living with tetraplegia in the community, especially for its ability to detect functional use in the home environment. The aim of this study was to develop and validate a wearable vision-based system for measuring hand use in the home among individuals living with tetraplegia. Several deep learning algorithms for detecting functional hand-object interactions were developed and compared. The most accurate algorithm was used to extract measures of hand function from 65 h of unscripted video recorded at home by 20 participants with tetraplegia. These measures were: the percentage of interaction time over total recording time (Perc); the average duration of individual interactions (Dur); and the number of interactions per hour (Num). To demonstrate the clinical validity of the technology, egocentric measures were correlated with validated clinical assessments of hand function and independence (Graded Redefined Assessment of Strength, Sensibility and Prehension [GRASSP], Upper Extremity Motor Score [UEMS], and Spinal Cord Independent Measure [SCIM]). Hand-object interactions were automatically detected with a median F1-score of 0.80 (0.67-0.87). Our results demonstrated that higher UEMS and better prehension were related to greater time spent interacting, whereas higher SCIM and better hand sensation resulted in a higher number of interactions performed during the egocentric video recordings. For the first time, measures of hand function automatically estimated in an unconstrained environment in individuals with tetraplegia have been validated against internationally accepted measures of hand function. Future work will necessitate a formal evaluation of the reliability and responsiveness of the egocentric-based performance measures for hand use.

Tenodesis Grasp Detection in Egocentric Video

OBJECTIVE

Cervical spinal cord injury (cSCI) can impair motor function in the upper limbs. Video from wearable cameras (egocentric video) has the potential to provide monitoring of rehabilitation outcomes at home, but methods for automated analysis of this data are needed. Wrist flexion and extension are essential elements to track grasping strategies after cSCI, as they may reflect the use of the tenodesis grasp, a common compensatory strategy. However, there is no established method to evaluate wrist flexion and extension from egocentric video.

METHODS

We propose a machine-learning-based approach comprising three steps-hand detection, pose estimation, and arm orientation estimation-to estimate wrist angle data, leading to the detection of tenodesis grasp.

RESULTS

The hand detection in conjunction with the pose estimation algorithm correctly located wrist and index finger metacarpophalangeal coordinates in 63% and 76% of 15,319 annotated frames, respectively, extracted from egocentric videos of individuals with cSCI performing activities of daily living in a home simulation laboratory. The arm orientation algorithm had a mean absolute error of 2.76 +/- 0.39 degrees in 12,863 labeled frames. Using these estimates, the presence of a tenodesis grasp was correctly detected in 72% +/- 11% of frames in videos of 6 activities.

CONCLUSION

The results provided a clear indication of which participants relied on tenodesis grasp and which did not.

SIGNIFICANCE

This paradigm provides the first method that can enable clinicians and researchers to monitor the use of the tenodesis grasp by individuals with cSCI at home, with implications for remote therapeutic guidance.

Perspectives and recommendations of individuals with tetraplegia regarding wearable cameras for monitoring hand function at home: Insights from a community-based study

CONTEXT

Wearable cameras have great potential for producing novel outcome measures of upper limb (UL) function and guiding care in individuals with cervical spinal cord injury (cSCI) living in the community. However, little is known about the perspectives of individuals with cSCI on the potential adoption of this technology.

OBJECTIVE

To analyze feedback from individuals with cSCI regarding the use of wearable cameras to record daily activities at home, in order to define guidelines for improving the design of this technology and fostering its implementation to optimize UL rehabilitation.

DESIGN

Mixed-methods study.

PARTICIPANTS

Thirteen adults with cSCI C3-C8 AIS A-D impairment.

MEASURES

Interview including survey and semi-structured questions.

RESULTS

Participants felt that this technology can provide naturalistic information regarding hand use to clinicians and researchers, which in turn can lead to better assessments of UL function and optimized therapies. Participants described the technology as easy-to-use but often reported discomfort that prevented them from conducting long recordings of fully natural activities. Privacy concerns included the possibility to capture household members and personal information displayed on objects (e.g. smartphones).

CONCLUSION

We provide the first set of guidelines to help researchers and therapists understand which steps need to be taken to translate wearable cameras into outpatient care and community-based research for UL rehabilitation. These guidelines include miniaturized and easy-to-wear cameras, as well as multiple measures for preventing privacy concerns such as avoiding public spaces and providing control over the recordings (e.g. start and stop the recordings at any time, keep or delete a recording).

Capturing hand use of individuals with spinal cord injury at home using egocentric video: a feasibility study

STUDY DESIGN

Feasibility study.

OBJECTIVE

The objective of this study is to explore the feasibility of capturing egocentric (first person) video recordings in the home of individuals with cervical spinal cord injury (SCI) for hand function evaluation.

SETTING

Community-based study in Toronto, Ontario, Canada.

METHODS

Three participants with SCI recorded activities of daily living (ADLs) at home without the presence of a researcher. Information regarding recording characteristics and compliance was obtained as well as structured and semi-structured interviews involving privacy, usefulness, and usability. A video processing algorithm capable of detecting interactions between the hand and objects was applied to the home recordings.

RESULTS

In all, 98.58 ± 1.05% of the obtained footage was usable and included four to eight unique activities over a span of 3-7 days. The interaction detection algorithm yielded an F1 score of 0.75 ± 0.15.

CONCLUSIONS

Capturing ADLs using an egocentric camera in the home environment after SCI is feasible. Considerations regarding privacy, ease of use of the devices, and scheduling of recordings are provided.

Generalizability of Hand-Object Interaction Detection in Egocentric Video across Populations with Hand Impairment

Stroke survivors often experience unilateral sensorimotor impairment. The restoration of upper limb function is an important determinant of quality of life after stroke. Wearable technologies that can measure hand function at home are needed to assess the impact of new interventions. Egocentric cameras combined with computer vision algorithms have been proposed as a means to capture hand use in unconstrained environments, and have shown promising results in this application for individuals with cervical spinal cord injury (cSCI). The objective of this study was to examine the generalizability of this approach to individuals who have experienced a stroke. An egocentric camera was used to capture the hand use (hand-object interactions) of 6 stroke survivors performing daily tasks in a home simulation laboratory. The interaction detection classifier previously trained on 9 individuals with cSCI was applied to detect hand use in the stroke survivors. The processing pipeline consisted of hand detection, hand segmentation, feature extraction, and interaction detection. The resulting average F1 scores for affected and unaffected hands were 0.66 ± 0.25 and 0.80 ± 0.15, respectively, indicating that the approach is feasible and has the potential to generalize to stroke survivors. Using stroke-specific training data may further increase the accuracy obtained for the affected hand.

A wearable vision-based system for detecting hand-object interactions in individuals with cervical spinal cord injury: First results in the home environment

Cervical spinal cord injury (cSCI) causes the paralysis of upper and lower limbs and trunk, significantly reducing quality of life and community participation of the affected individuals. The functional use of the upper limbs is the top recovery priority of people with cSCI and wearable vision-based systems have recently been proposed to extract objective outcome measures that reflect hand function in a natural context. However, previous studies were conducted in a controlled environment and may not be indicative of the actual hand use of people with cSCI living in the community. Thus, we propose a deep learning algorithm for automatically detecting hand-object interactions in egocentric videos recorded by participants with cSCI during their daily activities at home. The proposed approach is able to detect hand-object interactions with good accuracy (F1-score up to 0.82), demonstrating the feasibility of this system in uncontrolled situations (e.g., unscripted activities and variable illumination). This result paves the way for the development of an automated tool for measuring hand function in people with cSCI living in the community.

Egocentric video: a new tool for capturing hand use of individuals with spinal cord injury at home

Background

Current upper extremity outcome measures for persons with cervical spinal cord injury (cSCI) lack the ability to directly collect quantitative information in home and community environments. A wearable first-person (egocentric) camera system is presented that aims to monitor functional hand use outside of clinical settings.

Methods

The system is based on computer vision algorithms that detect the hand, segment the hand outline, distinguish the user’s left or right hand, and detect functional interactions of the hand with objects during activities of daily living. The algorithm was evaluated using egocentric video recordings from 9 participants with cSCI, obtained in a home simulation laboratory. The system produces a binary hand-object interaction decision for each video frame, based on features reflecting motion cues of the hand, hand shape and colour characteristics of the scene.

Results

The output from the algorithm was compared with a manual labelling of the video, yielding F1-scores of 0.74 ± 0.15 for the left hand and 0.73 ± 0.15 for the right hand. From the resulting frame-by-frame binary data, functional hand use measures were extracted: the amount of total interaction as a percentage of testing time, the average duration of interactions in seconds, and the number of interactions per hour. Moderate and significant correlations were found when comparing these output measures to the results of the manual labelling, with ρ = 0.40, 0.54 and 0.55 respectively.

Conclusions

These results demonstrate the potential of a wearable egocentric camera for capturing quantitative measures of hand use at home.