A wearable vision-based system for detecting hand-object interactions in individuals with cervical spinal cord injury: First results in the home environment

Measuring Hand Use in the Home after Cervical Spinal Cord Injury Using Egocentric Video

Egocentric video has recently emerged as a potential solution for monitoring hand function in individuals living with tetraplegia in the community, especially for its ability to detect functional use in the home environment. The aim of this study was to develop and validate a wearable vision-based system for measuring hand use in the home among individuals living with tetraplegia. Several deep learning algorithms for detecting functional hand-object interactions were developed and compared. The most accurate algorithm was used to extract measures of hand function from 65 h of unscripted video recorded at home by 20 participants with tetraplegia. These measures were: the percentage of interaction time over total recording time (Perc); the average duration of individual interactions (Dur); and the number of interactions per hour (Num). To demonstrate the clinical validity of the technology, egocentric measures were correlated with validated clinical assessments of hand function and independence (Graded Redefined Assessment of Strength, Sensibility and Prehension [GRASSP], Upper Extremity Motor Score [UEMS], and Spinal Cord Independent Measure [SCIM]). Hand-object interactions were automatically detected with a median F1-score of 0.80 (0.67-0.87). Our results demonstrated that higher UEMS and better prehension were related to greater time spent interacting, whereas higher SCIM and better hand sensation resulted in a higher number of interactions performed during the egocentric video recordings. For the first time, measures of hand function automatically estimated in an unconstrained environment in individuals with tetraplegia have been validated against internationally accepted measures of hand function. Future work will necessitate a formal evaluation of the reliability and responsiveness of the egocentric-based performance measures for hand use.

Egocentric vision-based detection of surfaces: towards context-aware free-living digital biomarkers for gait and fall risk assessment

Background

Falls in older adults are a critical public health problem. As a means to assess fall risks, free-living digital biomarkers (FLDBs), including spatiotemporal gait measures, drawn from wearable inertial measurement unit (IMU) data have been investigated to identify those at high risk. Although gait-related FLDBs can be impacted by intrinsic (e.g., gait impairment) and/or environmental (e.g., walking surfaces) factors, their respective impacts have not been differentiated by the majority of free-living fall risk assessment methods. This may lead to the ambiguous interpretation of the subsequent FLDBs, and therefore, less precise intervention strategies to prevent falls.

Methods

With the aim of improving the interpretability of gait-related FLDBs and investigating the impact of environment on older adults’ gait, a vision-based framework was proposed to automatically detect the most common level walking surfaces. Using a belt-mounted camera and IMUs worn by fallers and non-fallers (mean age 73.6 yrs), a unique dataset (i.e., Multimodal Ambulatory Gait and Fall Risk Assessment in the Wild (MAGFRA-W)) was acquired. The frames and image patches attributed to nine participants’ gait were annotated: (a) outdoor terrains: pavement (asphalt, cement, outdoor bricks/tiles), gravel, grass/foliage, soil, snow/slush; and (b) indoor terrains: high-friction materials (e.g., carpet, laminated floor), wood, and tiles. A series of ConvNets were developed: EgoPlaceNet categorizes frames into indoor and outdoor; and EgoTerrainNet (with outdoor and indoor versions) detects the enclosed terrain type in patches. To improve the framework’s generalizability, an independent training dataset with 9,424 samples was curated from different databases including GTOS and MINC-2500, and used for pretrained models’ (e.g., MobileNetV2) fine-tuning.

Results

EgoPlaceNet detected outdoor and indoor scenes in MAGFRA-W with 97.36$$\%$$ % and 95.59$$\%$$ % (leave-one-subject-out) accuracies, respectively. EgoTerrainNet-Indoor and -Outdoor achieved high detection accuracies for pavement (87.63$$\%$$ % ), foliage (91.24$$\%$$ % ), gravel (95.12$$\%$$ % ), and high-friction materials (95.02$$\%$$ % ), which indicate the models’ high generalizabiliy.

Conclusions

Encouraging results suggest that the integration of wearable cameras and deep learning approaches can provide objective contextual information in an automated manner, towards context-aware FLDBs for gait and fall risk assessment in the wild.

Perspectives and recommendations of individuals with tetraplegia regarding wearable cameras for monitoring hand function at home: Insights from a community-based study

CONTEXT

Wearable cameras have great potential for producing novel outcome measures of upper limb (UL) function and guiding care in individuals with cervical spinal cord injury (cSCI) living in the community. However, little is known about the perspectives of individuals with cSCI on the potential adoption of this technology.

OBJECTIVE

To analyze feedback from individuals with cSCI regarding the use of wearable cameras to record daily activities at home, in order to define guidelines for improving the design of this technology and fostering its implementation to optimize UL rehabilitation.

DESIGN

Mixed-methods study.

PARTICIPANTS

Thirteen adults with cSCI C3-C8 AIS A-D impairment.

MEASURES

Interview including survey and semi-structured questions.

RESULTS

Participants felt that this technology can provide naturalistic information regarding hand use to clinicians and researchers, which in turn can lead to better assessments of UL function and optimized therapies. Participants described the technology as easy-to-use but often reported discomfort that prevented them from conducting long recordings of fully natural activities. Privacy concerns included the possibility to capture household members and personal information displayed on objects (e.g. smartphones).

CONCLUSION

We provide the first set of guidelines to help researchers and therapists understand which steps need to be taken to translate wearable cameras into outpatient care and community-based research for UL rehabilitation. These guidelines include miniaturized and easy-to-wear cameras, as well as multiple measures for preventing privacy concerns such as avoiding public spaces and providing control over the recordings (e.g. start and stop the recordings at any time, keep or delete a recording).