Reinforcement-learning-based dual-control methodology for complex nonlinear discrete-time systems with application to spark engine EGR operation

A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach.

A suite of robust controllers for the manipulation of microscale objects

A suite of novel robust controllers is introduced for the pickup operation of microscale objects in a microelectromechanical system (MEMS). In MEMS, adhesive, surface tension, friction, and van der Waals forces are dominant. Moreover, these forces are typically unknown. The proposed robust controller overcomes the unknown contact dynamics and ensures its performance in the presence of actuator constraints by assuming that the upper bounds on these forces are known. On the other hand, for the robust adaptive critic-based neural network (NN) controller, the unknown dynamic forces are estimated online. It consists of an action NN for compensating the unknown system dynamics and a critic NN for approximating a certain strategic utility function and tuning the action NN weights. By using the Lyapunov approach, the uniform ultimate boundedness of the closed-loop manipulation error is shown for all the controllers for the pickup task. To imitate a practical system, a few system states are considered to be unavailable due to the presence of measurement noise. An output feedback version of the adaptive NN controller is proposed by exploiting the separation principle through a high-gain observer design. The problem of measurement noise is also overcome by constructing a reference system. Simulation results are presented and compared to substantiate the theoretical conclusions.

Discrete-time CMAC NN control of feedback linearizable nonlinear systems under a persistence of excitation

The local structure of CMAC neural networks (NN) result in better and faster controllers for nonlinear dynamical systems. A CMAC neural network-based discrete-time controller which linearizes the unknown multiinput and multioutput (MIMO) nonlinear system through feedback is presented. Control action is defined in order to achieve tracking performance for this unknown nonlinear system. An efficient and localized weight addressing scheme for the CMAC NN's is described using an appropriate choice of the B-spline receptive field functions that form a basis. A uniform ultimate boundedness of the closed-loop system is given in the sense of Lyapunov using the persistency of excitation (PE) condition. Simulation results are shown to demonstrate the theoretical conclusions.

Reinforcement Learning Neural-Network-Based Controller for Nonlinear Discrete-Time Systems With Input Constraints

A novel adaptive-critic-based neural network (NN) controller in discrete time is designed to deliver a desired tracking performance for a class of nonlinear systems in the presence of actuator constraints. The constraints of the actuator are treated in the controller design as the saturation nonlinearity. The adaptive critic NN controller architecture based on state feedback includes two NNs: the critic NN is used to approximate the "strategic" utility function, whereas the action NN is employed to minimize both the strategic utility function and the unknown nonlinear dynamic estimation errors. The critic and action NN weight updates are derived by minimizing certain quadratic performance indexes. Using the Lyapunov approach and with novel weight updates, the uniformly ultimate boundedness of the closed-loop tracking error and weight estimates is shown in the presence of NN approximation errors and bounded unknown disturbances. The proposed NN controller works in the presence of multiple nonlinearities, unlike other schemes that normally approximate one nonlinearity. Moreover, the adaptive critic NN controller does not require an explicit offline training phase, and the NN weights can be initialized at zero or random. Simulation results justify the theoretical analysis.

Adaptive Critic Neural Network-Based Object Grasping Control Using a Three-Finger Gripper

Grasping of objects has been a challenging task for robots. The complex grasping task can be defined as object contact control and manipulation subtasks. In this paper, object contact control subtask is defined as the ability to follow a trajectory accurately by the fingers of a gripper. The object manipulation subtask is defined in terms of maintaining a predefined applied force by the fingers on the object. A sophisticated controller is necessary since the process of grasping an object without a priori knowledge of the object's size, texture, softness, gripper, and contact dynamics is rather difficult. Moreover, the object has to be secured accurately and considerably fast without damaging it. Since the gripper, contact dynamics, and the object properties are not typically known beforehand, an adaptive critic neural network (NN)-based hybrid position/force control scheme is introduced. The feedforward action generating NN in the adaptive critic NN controller compensates the nonlinear gripper and contact dynamics. The learning of the action generating NN is performed on-line based on a critic NN output signal. The controller ensures that a three-finger gripper tracks a desired trajectory while applying desired forces on the object for manipulation. Novel NN weight tuning updates are derived for the action generating and critic NNs so that Lyapunov-based stability analysis can be shown. Simulation results demonstrate that the proposed scheme successfully allows fingers of a gripper to secure objects without the knowledge of the underlying gripper and contact dynamics of the object compared to conventional schemes.

Synthesis of boronic acid analogues of alpha-amino acids by introducing side chains as electrophiles

A synthetic route has been developed which has allowed us to prepare novel alpha-aminoboronic acids as inhibitors of serine proteases. These compounds were prepared to study the roles of proteases in biological systems. This methodology affords alpha-aminoboronic acids with the general formula R'-NHCH(R)BO(2)-pinanediol, where R = -CH(2)CHF(2), -CH(2)CO(2)tBu, and -(CH(2))(2)CO(2)Me and R' = either H or C(O)R". The latter two compounds are the boronic acid analogues of the natural amino acids aspartic acid and glutamic acid with the side chain carboxylate protected as a tert-butyl or a methyl ester, respectively. Following acylation of the amino group, the side chain tert-butyl ester of boroaspartic acid was removed by treatment with TFA. Boroglutamic acid was obtained as the free boronic acid by hydrolysis with HCl. Prior syntheses of alpha-aminoboronic acids involve the initial addition of an organometallic reagent to a trialkyl borate ester. These conditions do not allow the preparation of compounds with functionalities that are not stable to the strongly basic reaction conditions. The methodology described here allows the preparation of alpha-aminoboronic acids by introducing side chains as electrophiles. This is particularly advantageous for side chains which are prone to elimination or unwanted enolate formation. Specifically, BrCH(2)CHF(2), BrCH(2)COO(t)Bu, and CH(2)=CHCOOMe were allowed to react with the stabilized anion of (phenylthio)methane boronate, PhSCH(2)BO(2)C(6)H(12), to give the substituted boronate. The substituted (phenylthio)methane boronate was converted to the corresponding sulfonium ion by treatment with methyl iodide and subsequently displaced with iodide. The alpha-iodo derivative was converted to the amine by conventional methods.

L-glutamate binding sites of parasitic nematodes: an association with ivermectin resistance?

Nematode membrane preparations contain high amounts of low-affinity specific L-glutamate binding sites. The numbers of these sites were increased in 2 isolates, one field-derived and the other laboratory-derived, of ivermectin-resistant Haemonchus contortus and a field isolate of ivermectin-resistant Telodorsagia circumcincta, when compared to control, drug-sensitive isolates. Specific [3H]ivermectin binding to these membrane preparations showed no differences between ivermectin-sensitive and resistant isolates and the number of ivermectin binding sites was approximately 100-fold less than the number of L-glutamate binding sites. Kinetic analysis of L-glutamate binding suggested the presence of at least 2 classes of binding site. L-Glutamate binding was blocked by ibotenic acid, kynurenic acid and beta-hydroxyaspartate, but not by ivermectin, argiopine, kainate, quisqualate or NMDA. Competition assays with ibotenic acid suggested that there were 2 distinct populations of glutamate binding sites and that the site with the lower affinity for ibotenate was upregulated in the ivermectin-resistant nematodes. In the field isolate of resistant H. contortus we found no coding changes in the cDNAs encoding glutamate-gated chloride channel subunits HG2, HG3 and HG4, nor were any changes in channel expression detected using subunit-specific antibodies. The low-affinity binding site is unlikely to be associated with the ivermectin receptor in these nematodes.

Ligand-gated chloride channel subunits encoded by the Haemonchus contortus and Ascaris suum orthologues of the Caenorhabditis elegans gbr-2 (avr-14) gene

The alternatively-spliced Caenorhabditis elegans gbr-2/avr-14 gene encodes two subunits of the nematode ligand-gated chloride channel family which forms an important molecular target for the avermectin and related anthelminthics. We used reverse transcriptase-polymerase chain reaction (RT-PCR) techniques to isolate cDNAs encoding the products of the gbr-2/avr-14 orthologues from the parasitic nematodes Haemonchus contortus and Ascaris suum. The predicted polypeptides possess all the characteristics of subunits of the ligand-gated chloride channels, sharing greater than 80% amino-acid identity with their counterparts in C. elegans and with partial sequences from the filarial species Onchocerca volvulus and Dirofilaria immitis. The pattern of alternative splicing of the gbr-2/avr-14 gene observed in C. elegans is conserved in H. contortus but may not be in A. suum. Affinity-purified anti-GBR-2 antibodies were used to study the expression of these subunits in adult worms and they reacted specifically with the nerve ring, the ventral and dorsal nerve cords, the anterior portion of the dorsal sub-lateral cord and motor-neuron commissures in H. contortus. Specific immunofluorescence of the nerve cords was confirmed in A. suum; isolated muscle cells did not react with the antibody.

Inhibition of placental estrone sulfatase activity and MCF-7 breast cancer cell proliferation by estrone-3-amino derivatives

Estrogen levels in breast tumors of post-menopausal women are as much as 10 times higher than in plasma, presumably due to in situ formation of estrogen. Several lines of evidence indicate that the major source of estrogen in breast cancer cells may be from conversion of estrone sulfate to estrone by the enzyme estrone sulfatase. Inhibitors of estrone sulfatase may thus be potential agents for the treatment of estrogen-dependent breast cancer. We designed and synthesized a series of estrone-3-amino derivatives as potential estrone sulfatase inhibitors. We tested the inhibitory potential of these compounds using human placental microsomes, which contain a substantial amount of estrone sulfatase activity. Several compounds in the series significantly inhibited estrone sulfatase activity of the human placental microsomes when present at 10 microM. The IC50 for the estrone-3-amino compounds ranged from 8.7 to 14.6 microM. We next tested the ability of the estrone-3-amino derivatives to inhibit growth of the estrogen-dependent MCF-7 breast cancer cell line. MCF-7 cells showed substantial proliferation in the presence of 100 nM estrone sulfate in estrogen-free media, indicating that the cells were capable of converting estrone sulfate into estrone. The proliferative effect of estrone sulfate (1 microM) was significantly blocked by the estrone-3-amino derivatives at 10 microM. The magnitude of MCF-7 cell inhibition resulting from treatment with the estrone-3 amino compounds was similar to or exceeded that of Danazol, but was less than the level resulting from treatment with estrone sulfamate. Using data from all of the compounds tested, inhibition of MCF-7 cell proliferation was positively correlated with inhibition of placental estrone sulfatase activity, suggesting that the reduction in cell growth was attributable to the blockade of sulfatase activity. In support of this, there was no relationship between inhibition of estrone sulfatase activity and inhibition of cell growth when the estrogen-independent cell line MDA-MB-231 was used. Our results indicate the possible utility of estrone-3-amino derivatives for inhibition of estrone sulfatase activity. Further, our data support the concept that estrone sulfatase inhibitors may be useful as therapeutic agents for estrogen-dependent breast cancers.

Reversal of scopolamine induced amnesia in rats by the steroid sulfatase inhibitor estrone-3-O-sulfamate

The intent of the study was to determine whether altering the metabolism of neurosteroids via blockade of the enzyme, steroid sulfatase, could enhance retention test performance in rats. The steroid sulfatase inhibitor estrone-3-O-sulfamate (EMATE) was administered alone and in combination with the neurosteroid dehydroepiandrosterone sulfate (DHEAS) to rats which were then tested for the reversal of scopolamine induced amnesia. EMATE enhanced the reversal of amnesia by DHEAS as measured by a passive avoidance test. When administered without DHEAS, as a single acute dose, EMATE had no effect. When administered without DHEAS over 10 consecutive days, however, EMATE significantly improved retention. These results suggest that steroid sulfatase inhibition can potentiate the memory enhancing properties of DHEAS. The study also suggests that increasing the levels of endogenous sulfated neurosteroids via the inhibition of steroid sulfatase activity may enhance learning and/or memory function.

The subcostal nerve (ouch!): anatomical awareness in Salter's innominate osteotomy

The identification of the lateral cutaneous branch of the subcostal nerve supplying the anterior gluteal region while using the modified Smith-Petersen approach in performing Salter's innominate osteotomy has not been described previously. Following cadaveric dissection in six specimens and after performing routine Salter's innominate osteotomy using the modified Smith-Petersen approach in five patients, we found consistently that the lateral cutaneous branch of the subcostal nerve crossed the iliac crest between 2 and 5 cm posterior to the anterior superior iliac spine. This important accidental finding has led us to take care in identifying and protecting this nerve from any undue injury that could cause blunting of sensation in the anterior gluteal region.

Pes planus in childhood due to tibialis posterior tendon injuries. Treatment by flexor hallucis longus tendon transfer

Flat foot due to rupture of the tibialis posterior tendon has not previously been described in children. We present three young patients who developed unilateral pes planus after old undiagnosed lacerations of the tendon. Transfer of the flexor hallucis longus to the distal stump of the tibialis posterior tendon achieved good results in all three cases.