Application of Reinforcement Learning to Deep Brain Stimulation in a Computational Model of Parkinson's Disease

A simulation study of transcranial magnetoacoustic stimulation of the basal ganglia thalamic neural network to improve pathological beta oscillations in Parkinson's disease

BACKGROUND

Parkinson's disease (PD) is a common neurodegenerative disease. Transcranial magnetoacoustic stimulation (TMAS) is a new therapy that combines a transcranial focused acoustic pressure field with a magnetic field to excite or inhibit neurons in targeted area, which suppresses the abnormally elevated beta band amplitude in PD states, with high spatial resolution and non-invasively.

OBJECTIVE

To study the effective stimulation parameters of TMAS mononuclear and multinuclear stimulation for the treatment of PD with reduced beta band energy, improved abnormal synchronization, and no thermal damage.

METHODS

The TMAS model is constructed based on the volunteer's computed tomography, 128 arrays of phase-controlled transducers, and permanent magnets. A basal ganglia-thalamic (BG-Th) neural network model of the PD state was constructed on the basis of the Izhikevich model and the acoustic model. An ultrasound stimulation neuron model is constructed based on the Hodgkin-Huxley model. Numerical simulations of transcranial focused acoustic pressure field, temperature field and induced electric field at single and dual targets were performed using the locations of STN, GPi, and GPe in the human brain as the main stimulation target areas. And the acoustic and electric parameters at the focus were extracted to stimulate mononuclear and multinuclear in the BG-Th neural network.

RESULTS

When the stimulating effect of ultrasound is ignored, TMAS-STN simultaneously inhibits the beta-band amplitude of the GPi nucleus, whereas TMAS-GPi fails to simultaneously have an inhibitory effect on the STN. TMAS-STN&GPi can reduce the beta band amplitude. TMAS-STN&GPi&GPe suppressed the PD pathologic beta band amplitude of each nucleus to a greater extent. When considering the stimulatory effect of ultrasound, lower sound pressures of ultrasound do not affect the neuronal firing state, but higher sound pressures may promote or inhibit the stimulatory effect of induced currents.

CONCLUSIONS

At 9 T static magnetic field, 0.5-1.5 MPa and 1.5-2.0 MPa ultrasound had synergistic effects on individual STN and GPi neurons. TMAS multinuclear stimulation with appropriate ultrasound intensity was the most effective in suppressing the amplitude of pathological beta oscillations in PD and may be clinically useful.

Brain-machine interactive neuromodulation research tool with edge AI computing

Closed-loop neuromodulation with intelligence methods has shown great potentials in providing novel neuro-technology for treating neurological and psychiatric diseases. Development of brain-machine interactive neuromodulation strategies could lead to breakthroughs in precision and personalized electronic medicine. The neuromodulation research tool integrating artificial intelligent computing and performing neural sensing and stimulation in real-time could accelerate the development of closed-loop neuromodulation strategies and translational research into clinical application. In this study, we developed a brain-machine interactive neuromodulation research tool (BMINT), which has capabilities of neurophysiological signals sensing, computing with mainstream machine learning algorithms and delivering electrical stimulation pulse by pulse in real-time. The BMINT research tool achieved system time delay under 3 ms, and computing capabilities in feasible computation cost, efficient deployment of machine learning algorithms and acceleration process. Intelligent computing framework embedded in the BMINT enable real-time closed-loop neuromodulation developed with mainstream AI ecosystem resources. The BMINT could provide timely contribution to accelerate the translational research of intelligent neuromodulation by integrating neural sensing, edge AI computing and stimulation with AI ecosystems.

Multi-timescale neuromodulation strategy for closed-loop deep brain stimulation in Parkinson's disease

Objective.Beta triggered closed-loop deep brain stimulation (DBS) shows great potential for improving the efficacy while reducing side effect for Parkinson's disease. However, there remain great challenges due to the dynamics and stochasticity of neural activities. In this study, we aimed to tune the amplitude of beta oscillations with different time scales taking into account influence of inherent variations in the basal ganglia-thalamus-cortical circuit.Approach. A dynamic basal ganglia-thalamus-cortical mean-field model was established to emulate the medication rhythm. Then, a dynamic target model was designed to embody the multi-timescale dynamic of beta power with milliseconds, seconds and minutes. Moreover, we proposed a closed-loop DBS strategy based on a proportional-integral-differential (PID) controller with the dynamic control target. In addition, the bounds of stimulation amplitude increments and different parameters of the dynamic target were considered to meet the clinical constraints. The performance of the proposed closed-loop strategy, including beta power modulation accuracy, mean stimulation amplitude, and stimulation variation were calculated to determine the PID parameters and evaluate neuromodulation performance in the computational dynamic mean-field model.Main results. The Results show that the dynamic basal ganglia-thalamus-cortical mean-field model simulated the medication rhythm with the fasted and the slowest rate. The dynamic control target reflected the temporal variation in beta power from milliseconds to minutes. With the proposed closed-loop strategy, the beta power tracked the dynamic target with a smoother stimulation sequence compared with closed-loop DBS with the constant target. Furthermore, the beta power could be modulated to track the control target under different long-term targets, modulation strengths, and bounds of the stimulation increment.Significance. This work provides a new method of closed-loop DBS for multi-timescale beta power modulation with clinical constraints.

On-Off and Proportional Closed-Loop Adaptive Deep Brain Stimulation Reduces Motor Symptoms in Freely Moving Hemiparkinsonian Rats

OBJECTIVES

Closed-loop adaptive deep brain stimulation (aDBS) continuously adjusts stimulation parameters, with the potential to improve efficacy and reduce side effects of deep brain stimulation (DBS) for Parkinson's disease (PD). Rodent models can provide an effective platform for testing aDBS algorithms and establishing efficacy before clinical investigation. In this study, we compare two aDBS algorithms, on-off and proportional modulation of DBS amplitude, with conventional DBS in hemiparkinsonian rats.

MATERIALS AND METHODS

DBS of the subthalamic nucleus (STN) was delivered wirelessly in freely moving male and female hemiparkinsonian (N = 7) and sham (N = 3) Wistar rats. On-off and proportional aDBS, based on STN local field potential beta power, were compared with conventional DBS and three control stimulation algorithms. Behavior was assessed during cylinder tests (CT) and stepping tests (ST). Successful model creation was confirmed via apomorphine-induced rotation test and Tyrosine Hydroxylase-immunocytochemistry. Electrode location was histologically confirmed. Data were analyzed using linear mixed models.

RESULTS

Contralateral paw use in parkinsonian rats was reduced to 20% and 25% in CT and ST, respectively. Conventional, on-off, and proportional aDBS significantly improved motor function, restoring contralateral paw use to approximately 45% in both tests. No improvement in motor behavior was observed with either randomly applied on-off or low-amplitude continuous stimulation. Relative STN beta power was suppressed during DBS. Relative power in the alpha and gamma bands decreased and increased, respectively. Therapeutically effective adaptive DBS used approximately 40% less energy than did conventional DBS.

CONCLUSIONS

Adaptive DBS, using both on-off and proportional control schemes, is as effective as conventional DBS in reducing motor symptoms of PD in parkinsonian rats. Both aDBS algorithms yield substantial reductions in stimulation power. These findings support using hemiparkinsonian rats as a viable model for testing aDBS based on beta power and provide a path to investigate more complex closed-loop algorithms in freely behaving animals.

BGRL: Basal Ganglia inspired Reinforcement Learning based framework for deep brain stimulators

Deep Brain Stimulation (DBS) is an implantable medical device used for electrical stimulation to treat neurological disorders. Traditional DBS devices provide fixed frequency pulses, but personalized adjustment of stimulation parameters is crucial for optimal treatment. This paper introduces a Basal Ganglia inspired Reinforcement Learning (BGRL) approach, incorporating a closed-loop feedback mechanism to suppress neural synchrony during neurological fluctuations. The BGRL approach leverages the resemblance between the Basal Ganglia region of brain by incorporating the actor-critic architecture of reinforcement learning (RL). Simulation results demonstrate that BGRL significantly reduces synchronous electrical pulses compared to other standard RL algorithms. BGRL algorithm outperforms existing RL methods in terms of suppression capability and energy consumption, validated through comparisons using ensemble oscillators. Results shown in the paper demonstrate BGRL suppressed the synchronous electrical pulses across three signaling regimes namely regular, chaotic and bursting by 40%, 146% and 40% respectively as compared to soft actor-critic model. BGRL shows promise in effectively suppressing neural synchrony in DBS therapy, providing an efficient alternative to open-loop methodologies.

Machine learning for adaptive deep brain stimulation in Parkinson's disease: closing the loop

Parkinson's disease (PD) is the second most common neurodegenerative disease bearing a severe social and economic impact. So far, there is no known disease modifying therapy and the current available treatments are symptom oriented. Deep Brain Stimulation (DBS) is established as an effective treatment for PD, however current systems lag behind today's technological potential. Adaptive DBS, where stimulation parameters depend on the patient's physiological state, emerges as an important step towards "smart" DBS, a strategy that enables adaptive stimulation and personalized therapy. This new strategy is facilitated by currently available neurotechnologies allowing the simultaneous monitoring of multiple signals, providing relevant physiological information. Advanced computational models and analytical methods are an important tool to explore the richness of the available data and identify signal properties to close the loop in DBS. To tackle this challenge, machine learning (ML) methods applied to DBS have gained popularity due to their ability to make good predictions in the presence of multiple variables and subtle patterns. ML based approaches are being explored at different fronts such as the identification of electrophysiological biomarkers and the development of personalized control systems, leading to effective symptom relief. In this review, we explore how ML can help overcome the challenges in the development of closed-loop DBS, particularly its role in the search for effective electrophysiology biomarkers. Promising results demonstrate ML potential for supporting a new generation of adaptive DBS, with better management of stimulation delivery, resulting in more efficient and patient-tailored treatments.

Design and Simulation of a Passive Absorber to Reduce Measured Postural Tremor Signal

Tremor is a semi-rhythmic oscillatory movement of a body part caused by alternating simultaneous contractions of an antagonistic muscle group. Medical and surgical treatments used to reduce the symptoms of involuntary tremor causes negative side effects. This study examines the ability of passive vibration absorbers in reducing the amplitude of postural tremor (PT) type of involuntary tremors. An inertial measurement unit (IMU) is used to record PT signals at the forearm and hand of a patient. IMU signal is used as the active excitation input of an upper limb system modeled to represent the flexion-extension vibrational motion at the joints. Equations of motion are solved numerically to obtain a simulated response that fits the measured tremor signal. passive tuned mass damper (TMD) is modeled as a thin lightweight cantilever beam with a screw located at the position reflecting its operating frequency. Natural frequency of the TMD is derived for different screw positions and validated numerically and experimentally. Modal damping ratio of the TMD for each screw position is also estimated. Optimization of screw position and damping coefficient of the TMD depends on the minimization of the angular displacement amplitude at the wrist joint. A lightweight optimized three-TMD system of 28.64 g total effective mass, simulated using its estimated modal damping ratios, shows its effectiveness compared to the literature, in reducing 65-83% of the amplitudes at the joints. An experimental arm is prepared for further experimental validation before the design of a wearable anti-vibration bracelet.

The Influence of Transcranial Magnetoacoustic Stimulation Parameters on the Basal Ganglia-Thalamus Neural Network in Parkinson's Disease

Objective: Parkinson’s disease (PD) is a degenerative disease of the nervous system that frequently occurs in the aged. Transcranial magnetoacoustic stimulation (TMAS) is a neuronal adjustment method that combines sound fields and magnetic fields. It has the characteristics of high spatial resolution and noninvasive deep brain focusing.

Methods: This paper constructed a simulation model of TMAS based on volunteer’s skull computer tomography, phased controlled transducer and permanent magnet. It simulates a transcranial focused sound pressure field with the Westervelt equation and builds a basal ganglia and thalamus neural network model in the PD state based on the Hodgkin-Huxley model.

Results: A biased sinusoidal pulsed ultrasonic TMAS induced current with 0.3 T static magnetic field induction and 0.2 W⋅cm^–2 sound intensity can effectively modulate PD states with RI ≥ 0.633. The magnitude of magnetic induction strength was changed to 0.2 and 0.4 T. The induced current was the same when the sound intensity was 0.4 and 0.1 W⋅cm^–2. And the sound pressure level is in the range of −1 dB (the induced current difference is less than or equal to 0.019 μA⋅cm^–2). TMAS with a duty cycle of approximately 50% can effectively modulates the error firings in the PD neural network with a relay reliability not less than 0.633.

Conclusion: TMAS can modulates the state of PD.

Neuro4PD: An Initial Neurorobotics Model of Parkinson's Disease

In this work, we present the first steps toward the creation of a new neurorobotics model of Parkinson's Disease (PD) that embeds, for the first time in a real robot, a well-established computational model of PD. PD mostly affects the modulation of movement in humans. The number of people suffering from this neurodegenerative disease is set to double in the next 15 years and there is still no cure. With the new model we were capable to further explore the dynamics of the disease using a humanoid robot. Results show that the embedded model under both conditions, healthy and parkinsonian, was capable of performing a simple behavioural task with different levels of motor disturbance. We believe that this neurorobotics model is a stepping stone to the development of more sophisticated models that could eventually test and inform new PD therapies and help to reduce and replace animals in research.

FPGA-Based Real-Time Simulation Platform for Large-Scale STN-GPe Network

The real-time simulation of large-scale subthalamic nucleus (STN)-external globus pallidus (GPe) network model is of great significance for the mechanism analysis and performance improvement of deep brain stimulation (DBS) for Parkinson's states. This paper implements the real-time simulation of a large-scale STN-GPe network containing 512 single-compartment Hodgkin-Huxley type neurons on the Altera Stratix IV field programmable gate array (FPGA) hardware platform. At the single neuron level, some resource optimization schemes such as multiplier substitution, fixed-point operation, nonlinear function approximation and function recombination are adopted, which consists the foundation of the large-scale network realization. At the network level, the simulation scale of network is expanded using module reuse method at the cost of simulation time. The correlation coefficient between the neuron firing waveform of the FPGA platform and the MATLAB software simulation waveform is 0.9756. Under the same physiological time, the simulation speed of FPGA platform is 75 times faster than the Intel Core i7-8700K 3.70 GHz CPU 32GB RAM computer simulation speed. In addition, the established platform is used to analyze the effects of temporal pattern DBS on network firing activities. The proposed large-scale STN-GPe network meets the need of real time simulation, which would be rather helpful in designing closed-loop DBS improvement strategies.