Inferring Interaction Force from Visual Information without Using Physical Force Sensors

Machine Learning Approaches for 3D Motion Synthesis and Musculoskeletal Dynamics Estimation: A Survey

The inference of 3D motion and dynamics of the human musculoskeletal system has traditionally been solved using physics-based methods that exploit physical parameters to provide realistic simulations. Yet, such methods suffer from computational complexity and reduced stability, hindering their use in computer graphics applications that require real-time performance. With the recent explosion of data capture (mocap, video) machine learning (ML) has started to become popular as it is able to create surrogate models harnessing the huge amount of data stemming from various sources, minimizing computational time (instead of resource usage), and most importantly, approximate real-time solutions. The main purpose of this paper is to provide a review and classification of the most recent works regarding motion prediction, motion synthesis as well as musculoskeletal dynamics estimation problems using ML techniques, in order to offer sufficient insight into the state-of-the-art and draw new research directions. While the study of motion may appear distinct to musculoskeletal dynamics, these application domains provide jointly the link for more natural computer graphics character animation, since ML-based musculoskeletal dynamics estimation enables modeling of more long-term, temporally evolving, ergonomic effects, while offering automated and fast solutions. Overall, our review offers an in-depth presentation and classification of ML applications in human motion analysis, unlike previous survey articles focusing on specific aspects of motion prediction.

A Transparent Teleoperated Robotic Surgical System with Predictive Haptic Feedback and Force Modelling

In recent years, robotic minimally invasive surgery has transformed many types of surgical procedures and improved their outcomes. Implementing effective haptic feedback into a teleoperated robotic surgical system presents a significant challenge due to the trade-off between transparency and stability caused by system communication time delays. In this paper, these time delays are mitigated by implementing an environment estimation and force prediction methodology into an experimental robotic minimally invasive surgical system. At the slave, an exponentially weighted recursive least squares (EWRLS) algorithm estimates the respective parameters of the Kelvin-Voigt (KV) and Hunt-Crossley (HC) force models. The master then provides force feedback by interacting with a virtual environment via the estimated parameters. Palpation experiments were conducted with the slave in contact with polyurethane foam during human-in-the-loop teleoperation. The experimental results indicated that the prediction RMSE of error between predicted master force feedback and measured slave force was reduced to 0.076 N for the Hunt-Crossley virtual environment, compared to 0.356 N for the Kelvin-Voigt virtual environment and 0.560 N for the direct force feedback methodology. The results also demonstrated that the HC force model is well suited to provide accurate haptic feedback, particularly when there is a delay between the master and slave kinematics. Furthermore, a haptic feedback approach that incorporates environment estimation and force prediction improve transparency during teleoperation. In conclusion, the proposed bilateral master-slave robotic system has the potential to provide transparent and stable haptic feedback to the surgeon in surgical robotics procedures.

A Force-Feedback Methodology for Teleoperated Suturing Task in Robotic-Assisted Minimally Invasive Surgery

With robotic-assisted minimally invasive surgery (RAMIS), patients and surgeons benefit from a reduced incision size and dexterous instruments. However, current robotic surgery platforms lack haptic feedback, which is an essential element of safe operation. Moreover, teleportation control challenges make complex surgical tasks like suturing more time-consuming than those that use manual tools. This paper presents a new force-sensing instrument that semi-automates the suturing task and facilitates teleoperated robotic manipulation. In order to generate the ideal needle insertion trajectory and pass the needle through its curvature, the end-effector mechanism has a rotating degree of freedom. Impedance control was used to provide sensory information about needle-tissue interaction forces to the operator using an indirect force estimation approach based on data-based models. The operator's motion commands were then regulated using a hyperplanar virtual fixture (VF) designed to maintain the desired distance between the end-effector and tissue surface while avoiding unwanted contact. To construct the geometry of the VF, an optoelectronic sensor-based approach was developed. Based on the experimental investigation of the hyperplane VF methodology, improved needle-tissue interaction force, manipulation accuracy, and task completion times were demonstrated. Finally, experimental validation of the trained force estimation models and the perceived interaction forces by the user was conducted using online data, demonstrating the potential of the developed approach in improving task performance.

Cross-Modal Reconstruction for Tactile Signal in Human-Robot Interaction

A human can infer the magnitude of interaction force solely based on visual information because of prior knowledge in human-robot interaction (HRI). A method of reconstructing tactile information through cross-modal signal processing is proposed in this paper. In our method, visual information is added as an auxiliary source to tactile information. In this case, the receiver is only able to determine the tactile interaction force from the visual information provided. In our method, we first process groups of pictures (GOPs) and treat them as the input. Secondly, we use the low-rank foreground-based attention mechanism (LAM) to detect regions of interest (ROIs). Finally, we propose a linear regression convolutional neural network (LRCNN) to infer contact force in video frames. The experimental results show that our cross-modal reconstruction is indeed feasible. Furthermore, compared to other work, our method is able to reduce the complexity of the network and improve the material identification accuracy.

Force sensing in robot-assisted keyhole endoscopy: A systematic survey

Instrument–tissue interaction forces in minimally invasive surgery (MIS) provide valuable information that can be used to provide haptic perception, monitor tissue trauma, develop training guidelines, and evaluate the skill level of novice and expert surgeons. Force and tactile sensing is lost in many robot-assisted surgery (RAS) systems. Therefore, many researchers have focused on recovering this information through sensing systems and estimation algorithms. This article provides a comprehensive systematic review of the current force sensing research aimed at RAS and, more generally, keyhole endoscopy, in which instruments enter the body through small incisions. Articles published between January 2011 and May 2020 are considered, following the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. The literature search resulted in 110 papers on different force estimation algorithms and sensing technologies, sensor design specifications, and fabrication techniques.

Vision-Based Suture Tensile Force Estimation in Robotic Surgery

Compared to laparoscopy, robotics-assisted minimally invasive surgery has the problem of an absence of force feedback, which is important to prevent a breakage of the suture. To overcome this problem, surgeons infer the suture force from their proprioception and 2D image by comparing them to the training experience. Based on this idea, a deep-learning-based method using a single image and robot position to estimate the tensile force of the sutures without a force sensor is proposed. A neural network structure with a modified Inception Resnet-V2 and Long Short Term Memory (LSTM) networks is used to estimate the suture pulling force. The feasibility of proposed network is verified using the generated DB, recording the interaction under the condition of two different artificial skins and two different situations (in vivo and in vitro) at 13 viewing angles of the images by changing the tool positions collected from the master-slave robotic system. From the evaluation conducted to show the feasibility of the interaction force estimation, the proposed learning models successfully estimated the tensile force at 10 unseen viewing angles during training.

Dynamic-Vision-Based Force Measurements Using Convolutional Recurrent Neural Networks

In this paper, a novel dynamic Vision-Based Measurement method is proposed to measure contact force independent of the object sizes. A neuromorphic camera (Dynamic Vision Sensor) is utilizused to observe intensity changes within the silicone membrane where the object is in contact. Three deep Long Short-Term Memory neural networks combined with convolutional layers are developed and implemented to estimate the contact force from intensity changes over time. Thirty-five experiments are conducted using three objects with different sizes to validate the proposed approach. We demonstrate that the networks with memory gates are robust against variable contact sizes as the networks learn object sizes in the early stage of a grasp. Moreover, spatial and temporal features enable the sensor to estimate the contact force every 10 ms accurately. The results are promising with Mean Squared Error of less than 0.1 N for grasping and holding contact force using leave-one-out cross-validation method.

Future-Frame Prediction for Fast-Moving Objects with Motion Blur

We propose a deep neural network model that recognizes the position and velocity of a fast-moving object in a video sequence and predicts the object’s future motion. When filming a fast-moving subject using a regular camera rather than a super-high-speed camera, there is often severe motion blur, making it difficult to recognize the exact location and speed of the object in the video. Additionally, because the fast moving object usually moves rapidly out of the camera’s field of view, the number of captured frames used as input for future-motion predictions should be minimized. Our model can capture a short video sequence of two frames with a high-speed moving object as input, use motion blur as additional information to recognize the position and velocity of the object, and predict the video frame containing the future motion of the object. Experiments show that our model has significantly better performance than existing future-frame prediction models in determining the future position and velocity of an object in two physical scenarios where a fast-moving two-dimensional object appears.

Haptics in Teleoperated Medical Interventions: Force Measurement, Haptic Interfaces and Their Influence on User's Performance

OBJECTIVES

Haptics in teleoperated medical interventions enables measurement and transfer of force information to the operator during robot-environment interaction. This paper provides an overview of the current research in this domain and guidelines for future investigations.

METHODS

We review current technologies in force measurement and haptic devices as well as their experimental evaluation and influence on user's performance.

RESULTS

Force sensing is moving away from the conventional proximal measurement methods to distal sensing and contact-less methods. Wearable devices that deliver haptic feedback on different body parts are increasingly playing an important role. Performance and accuracy improvement are the widely reported benefits of haptic feedback, while there is a debate on its effect on task completion time and exerted force.

CONCLUSION

With the surge of new ideas, there is a need for better and more systematic validation of the new sensing and feedback technology, through better user studies and novel methods like validated benchmarks and new taxonomies.

SIGNIFICANCE

This review investigates haptics from sensing to interfaces within the context of user's performance and the validation procedures to highlight salient advances. It provides guidelines to future developments and highlights the shortcomings in the field.

Knocking and Listening: Learning Mechanical Impulse Response for Understanding Surface Characteristics

Inspired by spiders that can generate and sense vibrations to obtain information regarding a substrate, we propose an intelligent system that can recognize the type of surface being touched by knocking the surface and listening to the vibrations. Hence, we developed a system that is equipped with an electromagnetic hammer for hitting the ground and an accelerometer for measuring the mechanical responses induced by the impact. We investigate the feasibility of sensing 10 different daily surfaces through various machine-learning techniques including recent deep-learning approaches. Although some test surfaces are similar, experimental results show that our system can recognize 10 different surfaces remarkably well (test accuracy of 98.66%). In addition, our results without directly hitting the surface (internal impact) exhibited considerably high test accuracy (97.51%). Finally, we conclude this paper with the limitations and future directions of the study.

A Clamping Force Estimation Method Based on a Joint Torque Disturbance Observer Using PSO-BPNN for Cable-Driven Surgical Robot End-Effectors

The ability to sense external force is an important technique for force feedback, haptics and safe interaction control in minimally-invasive surgical robots (MISRs). Moreover, this ability plays a significant role in the restricting refined surgical operations. The wrist joints of surgical robot end-effectors are usually actuated by several long-distance wire cables. Its two forceps are each actuated by two cables. The scope of force sensing includes multidimensional external force and one-dimensional clamping force. This paper focuses on one-dimensional clamping force sensing method that do not require any internal force sensor integrated in the end-effector's forceps. A new clamping force estimation method is proposed based on a joint torque disturbance observer (JTDO) for a cable-driven surgical robot end-effector. The JTDO essentially considers the variations in cable tension between the actual cable tension and the estimated cable tension using a Particle Swarm Optimization Back Propagation Neural Network (PSO-BPNN) under free motion. Furthermore, a clamping force estimator is proposed based on the forceps' JTDO and their mechanical relations. According to comparative analyses in experimental studies, the detection resolutions of collision force and clamping force were 0.11 N. The experimental results verify the feasibility and effectiveness of the proposed clamping force sensing method.

Transparent and Flexible Mayan-Pyramid-based Pressure Sensor using Facile-Transferred Indium tin Oxide for Bimodal Sensor Applications

Transparent and conducting flexible electrodes have been successfully developed over the last few decades due to their potential applications in optoelectronics. However, recent developments in smart electronics, such as a direct human-machine interface, health-monitoring devices, motion-tracking sensors, and artificially electronic skin also require materials with multifunctional properties such as transparency, flexibility and good portability. In such devices, there remains room to develop transparent and flexible devices such as pressure sensors or temperature sensors. Herein, we demonstrate a fully transparent and flexible bimodal sensor using indium tin oxide (ITO), which is embedded in a plastic substrate. For the proposed pressure sensor, the embedded ITO is detached from its Mayan-pyramid-structured silicon mold by an environmentally friendly method which utilizes water-soluble sacrificial layers. The Mayan-pyramid-based pressure sensor is capable of six different pressure sensations with excellent sensitivity in the range of 100 Pa-10 kPa, high endurance of 105 cycles, and good pulse detection and tactile sensing data processing capabilities through machine learning (ML) algorithms for different surface textures. A 5 × 5-pixel pressure-temperature-based bimodal sensor array with a zigzag-shaped ITO temperature sensor on top of it is also demonstrated without a noticeable interface effect. This work demonstrates the potential to develop transparent bimodal sensors that can be employed for electronic skin (E-skin) applications.

An Efficient Three-Dimensional Convolutional Neural Network for Inferring Physical Interaction Force from Video

Interaction forces are traditionally predicted by a contact type haptic sensor. In this paper, we propose a novel and practical method for inferring the interaction forces between two objects based only on video data-one of the non-contact type camera sensors-without the use of common haptic sensors. In detail, we could predict the interaction force by observing the texture changes of the target object by an external force. For this purpose, our hypothesis is that a three-dimensional (3D) convolutional neural network (CNN) can be made to predict the physical interaction forces from video images. In this paper, we proposed a bottleneck-based 3D depthwise separable CNN architecture where the video is disentangled into spatial and temporal information. By applying the basic depthwise convolution concept to each video frame, spatial information can be efficiently learned; for temporal information, the 3D pointwise convolution can be used to learn the linear combination among sequential frames. To validate and train the proposed model, we collected large quantities of datasets, which are video clips of the physical interactions between two objects under different conditions (illumination and angle variations) and the corresponding interaction forces measured by the haptic sensor (as the ground truth). Our experimental results confirmed our hypothesis; when compared with previous models, the proposed model was more accurate and efficient, and although its model size was 10 times smaller, the 3D convolutional neural network architecture exhibited better accuracy. The experiments demonstrate that the proposed model remains robust under different conditions and can successfully estimate the interaction force between objects.

Context-Aware Winter Sports Based on Multivariate Sequence Learning

In this paper, we present an intelligent system that is capable of estimating the status of a player engaging in winter activities based on the sequence analysis of multivariate time-series sensor signals. Among the winter activities, this paper mainly focuses on downhill winter sports such as alpine skiing and snowboarding. Assuming that the mechanical vibrations generated by physical interaction between the ground surface and ski/snowboard in motion can describe the ground conditions and playing contexts, we utilize inertial and vibration signals to categorize the motion context. For example, the proposed system estimates whether the player is sitting on a ski lift or standing on the escalator, or skiing on wet or snowy ground, etc. To measure the movement of a player during a game or on the move, we develop a custom embedded system comprising a motion sensor and piezo transducer. The captured multivariate sequence signals are then trained in a supervised fashion. We adopt artificial neural network approaches (e.g., 1D convolutional neural network, and gated recurrent neural networks, such as long short-term memory and gated recurrent units). The experimental results validate the feasibility of the proposed approach.

Design and Evaluation of FBG-Based Tension Sensor in Laparoscope Surgical Robots

Due to the narrow space and a harsh chemical environment in the sterilization processes for the end-effector of surgical robots, it is difficult to install and integrate suitable sensors for the purpose of effective and precise force control. This paper presents an innovative tension sensor for estimation of grasping force in our laparoscope surgical robot. The proposed sensor measures the tension of cable using fiber gratings (FBGs) which are pasted in the grooves on the inclined cantilevers of the sensor. By exploiting the stain measurement characteristics of FBGs, the small deformation of the inclined cantilevers caused by the cable tension can be measured. The working principle and the sensor model are analyzed. Based on the sensor model, the dimensions of the sensor are designed and optimized. A dedicated experimental setup is established to calibrate and test the sensor. The results of experiments for estimation the grasping force validate the sensor.