1
|
Slade P, Atkeson C, Donelan JM, Houdijk H, Ingraham KA, Kim M, Kong K, Poggensee KL, Riener R, Steinert M, Zhang J, Collins SH. On human-in-the-loop optimization of human-robot interaction. Nature 2024; 633:779-788. [PMID: 39322732 DOI: 10.1038/s41586-024-07697-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Accepted: 06/07/2024] [Indexed: 09/27/2024]
Abstract
From industrial exoskeletons to implantable medical devices, robots that interact closely with people are poised to improve every aspect of our lives. Yet designing these systems is very challenging; humans are incredibly complex and, in many cases, we respond to robotic devices in ways that cannot be modelled or predicted with sufficient accuracy. A new approach, human-in-the-loop optimization, can overcome these challenges by systematically and empirically identifying the device characteristics that result in the best objective performance for a specific user and application. This approach has enabled substantial improvements in human-robot performance in research settings and has the potential to speed development and enhance products. In this Perspective, we describe methods for applying human-in-the-loop optimization to new human-robot interaction problems, addressing each key decision in a variety of contexts. We also identify opportunities to develop new optimization techniques and answer underlying scientific questions. We anticipate that our readers will advance human-in-the-loop optimization and use it to design robotic devices that truly enhance the human experience.
Collapse
Affiliation(s)
- Patrick Slade
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA.
| | | | - J Maxwell Donelan
- WearTech Labs, Simon Fraser University, Burnaby, British Columbia, Canada
| | - Han Houdijk
- Department of Human Movement Sciences, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
| | - Kimberly A Ingraham
- Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA
| | - Myunghee Kim
- Mechanical and Industrial Engineering, University of Illinois Chicago, Chicago, IL, USA
| | - Kyoungchul Kong
- Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea
| | - Katherine L Poggensee
- Department of Rehabilitation Medicine, Erasmus Medical Center, Rotterdam, The Netherlands
- Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands
| | - Robert Riener
- Sensory-Motor Systems Lab, ETH Zurich, Zürich, Switzerland
- Faculty of Medicine, University of Zurich, Zürich, Switzerland
| | - Martin Steinert
- Department of Mechanical and Industrial Engineering, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| | - Juanjuan Zhang
- College of Artificial Intelligence, Institute of Robotics and Automatic Information System, Nankai University, Tianjin, China
| | - Steven H Collins
- Department of Mechanical Engineering, Stanford University, Stanford, CA, USA.
| |
Collapse
|
2
|
van der Grinten M, de Ruyter van Steveninck J, Lozano A, Pijnacker L, Rueckauer B, Roelfsema P, van Gerven M, van Wezel R, Güçlü U, Güçlütürk Y. Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses. eLife 2024; 13:e85812. [PMID: 38386406 PMCID: PMC10883675 DOI: 10.7554/elife.85812] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2022] [Accepted: 01/21/2024] [Indexed: 02/23/2024] Open
Abstract
Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or 'phosphenes') has limited resolution, and a great portion of the field's research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator's suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.
Collapse
Affiliation(s)
| | | | - Antonio Lozano
- Netherlands Institute for Neuroscience, Vrije Universiteit, Amsterdam, Netherlands
| | - Laura Pijnacker
- Donders Institute for Brain Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
| | - Bodo Rueckauer
- Donders Institute for Brain Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
| | - Pieter Roelfsema
- Netherlands Institute for Neuroscience, Vrije Universiteit, Amsterdam, Netherlands
| | - Marcel van Gerven
- Donders Institute for Brain Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
| | - Richard van Wezel
- Donders Institute for Brain Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
- Biomedical Signals and Systems Group, University of Twente, Enschede, Netherlands
| | - Umut Güçlü
- Donders Institute for Brain Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
| | - Yağmur Güçlütürk
- Donders Institute for Brain Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
| |
Collapse
|
3
|
Leong F, Rahmani B, Psaltis D, Moser C, Ghezzi D. An actor-model framework for visual sensory encoding. Nat Commun 2024; 15:808. [PMID: 38280912 PMCID: PMC10821921 DOI: 10.1038/s41467-024-45105-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Accepted: 01/15/2024] [Indexed: 01/29/2024] Open
Abstract
A fundamental challenge in neuroengineering is determining a proper artificial input to a sensory system that yields the desired perception. In neuroprosthetics, this process is known as artificial sensory encoding, and it holds a crucial role in prosthetic devices restoring sensory perception in individuals with disabilities. For example, in visual prostheses, one key aspect of artificial image encoding is to downsample images captured by a camera to a size matching the number of inputs and resolution of the prosthesis. Here, we show that downsampling an image using the inherent computation of the retinal network yields better performance compared to learning-free downsampling methods. We have validated a learning-based approach (actor-model framework) that exploits the signal transformation from photoreceptors to retinal ganglion cells measured in explanted mouse retinas. The actor-model framework generates downsampled images eliciting a neuronal response in-silico and ex-vivo with higher neuronal reliability than the one produced by a learning-free approach. During the learning process, the actor network learns to optimize contrast and the kernel's weights. This methodological approach might guide future artificial image encoding strategies for visual prostheses. Ultimately, this framework could be applicable for encoding strategies in other sensory prostheses such as cochlear or limb.
Collapse
Affiliation(s)
- Franklin Leong
- Medtronic Chair in Neuroengineering, Center for Neuroprosthetics and Institute of Bioengineering, School of Engineering, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Babak Rahmani
- Laboratory of Applied Photonics Devices, Institute of Electrical and Micro Engineering, School of Engineering, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
- Microsoft Research, Cambridge, UK
| | - Demetri Psaltis
- Optics Laboratory, Institute of Electrical and Micro Engineering, School of Engineering, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Christophe Moser
- Laboratory of Applied Photonics Devices, Institute of Electrical and Micro Engineering, School of Engineering, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Diego Ghezzi
- Medtronic Chair in Neuroengineering, Center for Neuroprosthetics and Institute of Bioengineering, School of Engineering, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland.
- Ophthalmic and Neural Technologies Laboratory, Department of Ophthalmology, University of Lausanne, Hôpital ophtalmique Jules-Gonin, Fondation Asile des Aveugles, Lausanne, Switzerland.
| |
Collapse
|
4
|
Mirzakhalili E, Rogers ER, Lempka SF. An optimization framework for targeted spinal cord stimulation. J Neural Eng 2023; 20:056026. [PMID: 37647885 PMCID: PMC10535048 DOI: 10.1088/1741-2552/acf522] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 08/14/2023] [Accepted: 08/30/2023] [Indexed: 09/01/2023]
Abstract
Objective. Spinal cord stimulation (SCS) is a common neurostimulation therapy to manage chronic pain. Technological advances have produced new neurostimulation systems with expanded capabilities in an attempt to improve the clinical outcomes associated with SCS. However, these expanded capabilities have dramatically increased the number of possible stimulation parameters and made it intractable to efficiently explore this large parameter space within the context of standard clinical programming procedures. Therefore, in this study, we developed an optimization approach to define the optimal current amplitudes or fractions across individual contacts in an SCS electrode array(s).Approach. We developed an analytic method using the Lagrange multiplier method along with smoothing approximations. To test our optimization framework, we used a hybrid computational modeling approach that consisted of a finite element method model and multi-compartment models of axons and cells within the spinal cord. Moreover, we extended our approach to multi-objective optimization to explore the trade-off between activating regions of interest (ROIs) and regions of avoidance (ROAs).Main results. For simple ROIs, our framework suggested optimized configurations that resembled simple bipolar configurations. However, when we considered multi-objective optimization, our framework suggested nontrivial stimulation configurations that could be selected from Pareto fronts to target multiple ROIs or avoid ROAs.Significance. We developed an optimization framework for targeted SCS. Our method is analytic, which allows for the fast calculation of optimal solutions. For the first time, we provided a multi-objective approach for selective SCS. Through this approach, we were able to show that novel configurations can provide neural recruitment profiles that are not possible with conventional stimulation configurations (e.g. bipolar stimulation). Most importantly, once integrated with computational models that account for sources of interpatient variability (e.g. anatomy, electrode placement), our optimization framework can be utilized to provide stimulation settings tailored to the needs of individual patients.
Collapse
Affiliation(s)
- Ehsan Mirzakhalili
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Biointerfaces Institute, University of Michigan, Ann Arbor, MI, United States of America
| | - Evan R Rogers
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Biointerfaces Institute, University of Michigan, Ann Arbor, MI, United States of America
| | - Scott F Lempka
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Biointerfaces Institute, University of Michigan, Ann Arbor, MI, United States of America
- Department of Anesthesiology, University of Michigan, Ann Arbor, MI, United States of America
| |
Collapse
|
5
|
Wu Y, Karetic I, Stegmaier J, Walter P, Merhof D. A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38082738 DOI: 10.1109/embc40787.2023.10340288] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
We propose a neural network-based framework to optimize the perceptions simulated by the in silico retinal implant model pulse2percept. The overall pipeline consists of a trainable encoder, a pre-trained retinal implant model and a pre-trained evaluator. The encoder is a U-Net, which takes the original image and outputs the stimulus. The pre-trained retinal implant model is also a U-Net, which is trained to mimic the biomimetic perceptual model implemented in pulse2percept. The evaluator is a shallow VGG classifier, which is trained with original images. Based on 10,000 test images from the MNIST dataset, we show that the convolutional neural network-based encoder performs significantly better than the trivial downsampling approach, yielding a boost in the weighted F1-Score by 36.17% in the pre-trained classifier with 6×10 electrodes. With this fully neural network-based encoder, the quality of the downstream perceptions can be fine-tuned using gradient descent in an end-to-end fashion.
Collapse
|
6
|
Beyeler M, Sanchez-Garcia M. Towards a Smart Bionic Eye: AI-powered artificial vision for the treatment of incurable blindness. J Neural Eng 2022; 19:10.1088/1741-2552/aca69d. [PMID: 36541463 PMCID: PMC10507809 DOI: 10.1088/1741-2552/aca69d] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2022] [Accepted: 11/28/2022] [Indexed: 11/30/2022]
Abstract
Objective.How can we return a functional form of sight to people who are living with incurable blindness? Despite recent advances in the development of visual neuroprostheses, the quality of current prosthetic vision is still rudimentary and does not differ much across different device technologies.Approach.Rather than aiming to represent the visual scene as naturally as possible, aSmart Bionic Eyecould provide visual augmentations through the means of artificial intelligence-based scene understanding, tailored to specific real-world tasks that are known to affect the quality of life of people who are blind, such as face recognition, outdoor navigation, and self-care.Main results.Complementary to existing research aiming to restore natural vision, we propose a patient-centered approach to incorporate deep learning-based visual augmentations into the next generation of devices.Significance.The ability of a visual prosthesis to support everyday tasks might make the difference between abandoned technology and a widely adopted next-generation neuroprosthetic device.
Collapse
Affiliation(s)
- Michael Beyeler
- Department of Computer Science,University of California,Santa Barbara, CA, United States of America
- Department of Psychological & Brain Sciences, University of California, Santa Barbara, CA, United States of America
| | - Melani Sanchez-Garcia
- Department of Computer Science,University of California,Santa Barbara, CA, United States of America
| |
Collapse
|