Clinical and in vitro application of robotic computer-assisted implant surgery: a scoping review

In recent years, the emergence and application of robotic computer-assisted implant surgery (r-CAIS) has resulted in a revolutionary shift in conventional implant diagnosis and treatment. This scoping review was performed to verify the null hypothesis that r-CAIS has a relatively high accuracy of within 1 mm, with relatively few complications and a short operative time. This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR). From the 3355 publications identified in the PubMed, Scopus, Web of Science, and Google Scholar databases, 28 were finally included after a comprehensive review and analysis. The null hypothesis is partly accepted, as r-CAIS has a relatively high accuracy (coronal and apical deviation within 1 mm), and no significant adverse events or complications have been reported to date, although additional confirmatory studies are needed. However, there is insufficient evidence for a shorter surgical time, and further clinical research on this topic is required.

Advances of surgical robotics: image-guided classification and application

Surgical robotics application in the field of minimally invasive surgery has developed rapidly and has been attracting increasingly more research attention in recent years. A common consensus has been reached that surgical procedures are to become less traumatic and with the implementation of more intelligence and higher autonomy, which is a serious challenge faced by the environmental sensing capabilities of robotic systems. One of the main sources of environmental information for robots are images, which are the basis of robot vision. In this review article, we divide clinical image into direct and indirect based on the object of information acquisition, and into continuous, intermittent continuous, and discontinuous according to the target-tracking frequency. The characteristics and applications of the existing surgical robots in each category are introduced based on these two dimensions. Our purpose in conducting this review was to analyze, summarize, and discuss the current evidence on the general rules on the application of image technologies for medical purposes. Our analysis gives insight and provides guidance conducive to the development of more advanced surgical robotics systems in the future.

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

BACKGROUND

Laparoscopic cholecystectomy is a very frequent surgical procedure. However, in an ageing society, less surgical staff will need to perform surgery on patients. Collaborative surgical robots (cobots) could address surgical staff shortages and workload. To achieve context-awareness for surgeon-robot collaboration, the intraoperative action workflow recognition is a key challenge.

METHODS

A surgical process model was developed for intraoperative surgical activities including actor, instrument, action and target in laparoscopic cholecystectomy (excluding camera guidance). These activities, as well as instrument presence and surgical phases were annotated in videos of laparoscopic cholecystectomy performed on human patients (n = 10) and on explanted porcine livers (n = 10). The machine learning algorithm Distilled-Swin was trained on our own annotated dataset and the CholecT45 dataset. The validation of the model was conducted using a fivefold cross-validation approach.

RESULTS

In total, 22,351 activities were annotated with a cumulative duration of 24.9 h of video segments. The machine learning algorithm trained and validated on our own dataset scored a mean average precision (mAP) of 25.7% and a top K = 5 accuracy of 85.3%. With training and validation on our dataset and CholecT45, the algorithm scored a mAP of 37.9%.

CONCLUSIONS

An activity model was developed and applied for the fine-granular annotation of laparoscopic cholecystectomies in two surgical settings. A machine recognition algorithm trained on our own annotated dataset and CholecT45 achieved a higher performance than training only on CholecT45 and can recognize frequently occurring activities well, but not infrequent activities. The analysis of an annotated dataset allowed for the quantification of the potential of collaborative surgical robots to address the workload of surgical staff. If collaborative surgical robots could grasp and hold tissue, up to 83.5% of the assistant's tissue interacting tasks (i.e. excluding camera guidance) could be performed by robots.

Evaluation of an Active Disturbance Rejection Controller for Ophthalmic Robots with Piezo-Driven Injector

Retinal vein cannulation involves puncturing an occluded vessel on the micron scale. Even single millinewton force can cause permanent damage. An ophthalmic robot with a piezo-driven injector is precise enough to perform this delicate procedure, but the uncertain viscoelastic characteristics of the vessel make it difficult to achieve the desired contact force without harming the retina. The paper utilizes a viscoelastic contact model to explain the mechanical characteristics of retinal blood vessels to address this issue. The uncertainty in the viscoelastic properties is considered an internal disturbance of the contact model, and an active disturbance rejection controller is then proposed to precisely control the contact force. The experimental results show that this method can precisely adjust the contact force at the millinewton level even when the viscoelastic parameters vary significantly (up to 403.8%). The root mean square (RMS) and maximum value of steady-state error are 0.32 mN and 0.41 mN. The response time is below 2.51 s with no obvious overshoot.

A fully autonomous robotic ultrasound system for thyroid scanning

The current thyroid ultrasound relies heavily on the experience and skills of the sonographer and the expertise of the radiologist, and the process is physically and cognitively exhausting. In this paper, we report a fully autonomous robotic ultrasound system, which is able to scan thyroid regions without human assistance and identify malignant nod- ules. In this system, human skeleton point recognition, reinforcement learning, and force feedback are used to deal with the difficulties in locating thyroid targets. The orientation of the ultrasound probe is adjusted dynamically via Bayesian optimization. Experimental results on human participants demonstrated that this system can perform high-quality ultrasound scans, close to manual scans obtained by clinicians. Additionally, it has the potential to detect thyroid nodules and provide data on nodule characteristics for American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) calculation.

Surgical outcomes of robotic versus conventional autologous breast reconstruction: a systematic review and meta-analysis

Breast reconstruction is an integral part of breast cancer management. Conventional techniques of flap harvesting for autologous breast reconstruction are associated with considerable complications. Robotic surgery has enabled a new spectrum of minimally invasive breast surgeries. The current systematic review and meta-analysis study was designed to retrieve the surgical and clinical outcomes of robotic versus conventional techniques for autologous breast reconstruction. An extensive systematic literature review was performed from inception to 25 April 2023. All clinical studies comparing the outcomes of robotic and conventional autologous breast reconstruction were included for meta-analysis. The present meta-analysis included seven articles consisting of 783 patients. Of them, 263 patients received robotic breast reconstruction, while 520 patients received conventional technique. Of note, 477 patients received latissimus dorsi flap (LDF) and 306 were subjected to deep inferior epigastric artery perforator (DIEP) flap. There was a significantly prolonged duration of surgery (MD 58.36;95% CI 32.05,84.67;P < 0.001) and duration of anaesthesia (MD 47;95% CI 16.23,77.77;P = 0.003) among patients who underwent robotic surgery. There was a similar risk of complications between robotic and conventional surgeries. The mean level of pain intensity was significantly lower among patients who received robotic breast surgery (MD- 0.28;95% CI - 0.73,0.17; P = 0.22). There was prolonged length of hospitalization among patients with conventional DIEP flap surgery (MD- 0.59;95% CI - 1.13,- 0.05;P = 0.03). The present meta-analysis highlighted the feasibility, safety, and effectiveness of robotic autologous breast reconstruction. This included the successful harvesting of LDF and DIEP flap with acceptable surgical and functional outcomes.

Applications of artificial intelligence in urologic oncology

PURPOSE

With the recent rising interest in artificial intelligence (AI) in medicine, many studies have explored the potential and usefulness of AI in urological diseases. This study aimed to comprehensively review recent applications of AI in urologic oncology.

MATERIALS AND METHODS

We searched the PubMed-MEDLINE databases for articles in English on machine learning (ML) and deep learning (DL) models related to general surgery and prostate, bladder, and kidney cancer. The search terms were a combination of keywords, including both "urology" and "artificial intelligence" with one of the following: "machine learning," "deep learning," "neural network," "renal cell carcinoma," "kidney cancer," "urothelial carcinoma," "bladder cancer," "prostate cancer," and "robotic surgery."

RESULTS

A total of 58 articles were included. The studies on prostate cancer were related to grade prediction, improved diagnosis, and predicting outcomes and recurrence. The studies on bladder cancer mainly used radiomics to identify aggressive tumors and predict treatment outcomes, recurrence, and survival rates. Most studies on the application of ML and DL in kidney cancer were focused on the differentiation of benign and malignant tumors as well as prediction of their grade and subtype. Most studies suggested that methods using AI may be better than or similar to existing traditional methods.

CONCLUSIONS

AI technology is actively being investigated in the field of urological cancers as a tool for diagnosis, prediction of prognosis, and decision-making and is expected to be applied in additional clinical areas soon. Despite technological, legal, and ethical concerns, AI will change the landscape of urological cancer management.

Levels of autonomy in FDA-cleared surgical robots: a systematic review

The integration of robotics in surgery has increased over the past decade, and advances in the autonomous capabilities of surgical robots have paralleled that of assistive and industrial robots. However, classification and regulatory frameworks have not kept pace with the increasing autonomy of surgical robots. There is a need to modernize our classification to understand technological trends and prepare to regulate and streamline surgical practice around these robotic systems. We present a systematic review of all surgical robots cleared by the United States Food and Drug Administration (FDA) from 2015 to 2023, utilizing a classification system that we call Levels of Autonomy in Surgical Robotics (LASR) to categorize each robot's decision-making and action-taking abilities from Level 1 (Robot Assistance) to Level 5 (Full Autonomy). We searched the 510(k), De Novo, and AccessGUDID databases in December 2023 and included all medical devices fitting our definition of a surgical robot. 37,981 records were screened to identify 49 surgical robots. Most surgical robots were at Level 1 (86%) and some reached Level 3 (Conditional Autonomy) (6%). 2 surgical robots were recognized by the FDA to have machine learning-enabled capabilities, while more were reported to have these capabilities in their marketing materials. Most surgical robots were introduced via the 510(k) pathway, but a growing number via the De Novo pathway. This review highlights trends toward greater autonomy in surgical robotics. Implementing regulatory frameworks that acknowledge varying levels of autonomy in surgical robots may help ensure their safe and effective integration into surgical practice.

Evolving Therapeutic Landscape of Intracerebral Hemorrhage: Emerging Cutting-Edge Advancements in Surgical Robots, Regenerative Medicine, and Neurorehabilitation Techniques

Intracerebral hemorrhage (ICH) is the most serious form of stroke and has limited available therapeutic options. As knowledge on ICH rapidly develops, cutting-edge techniques in the fields of surgical robots, regenerative medicine, and neurorehabilitation may revolutionize ICH treatment. However, these new advances still must be translated into clinical practice. In this review, we examined several emerging therapeutic strategies and their major challenges in managing ICH, with a particular focus on innovative therapies involving robot-assisted minimally invasive surgery, stem cell transplantation, in situ neuronal reprogramming, and brain-computer interfaces. Despite the limited expansion of the drug armamentarium for ICH over the past few decades, the judicious selection of more efficacious therapeutic modalities and the exploration of multimodal combination therapies represent opportunities to improve patient prognoses after ICH.

An Adaptive Human-Robotic Interaction Architecture for Augmenting Surgery Performance Using Real-Time Workload Sensing-Demonstration of a Semi-autonomous Suction Tool

OBJECTIVE

This study developed and evaluated a mental workload-based adaptive automation (MWL-AA) that monitors surgeon cognitive load and assist during cognitively demanding tasks and assists surgeons in robotic-assisted surgery (RAS).

BACKGROUND

The introduction of RAS makes operators overwhelmed. The need for precise, continuous assessment of human mental workload (MWL) states is important to identify when the interventions should be delivered to moderate operators' MWL.

METHOD

The MWL-AA presented in this study was a semi-autonomous suction tool. The first experiment recruited ten participants to perform surgical tasks under different MWL levels. The physiological responses were captured and used to develop a real-time multi-sensing model for MWL detection. The second experiment evaluated the effectiveness of the MWL-AA, where nine brand-new surgical trainees performed the surgical task with and without the MWL-AA. Mixed effect models were used to compare task performance, objective- and subjective-measured MWL.

RESULTS

The proposed system predicted high MWL hemorrhage conditions with an accuracy of 77.9%. For the MWL-AA evaluation, the surgeons' gaze behaviors and brain activities suggested lower perceived MWL with MWL-AA than without. This was further supported by lower self-reported MWL and better task performance in the task condition with MWL-AA.

CONCLUSION

A MWL-AA systems can reduce surgeons' workload and improve performance in a high-stress hemorrhaging scenario. Findings highlight the potential of utilizing MWL-AA to enhance the collaboration between the autonomous system and surgeons. Developing a robust and personalized MWL-AA is the first step that can be used do develop additional use cases in future studies.

APPLICATION

The proposed framework can be expanded and applied to more complex environments to improve human-robot collaboration.

And say the AI responded? Dancing around 'autonomy' in AI/human encounters

The article explores technology-human relations in a time of artificial intelligence (AI) and in the context of long-standing problems in social theory about agency, nonhumans, and autonomy. Most theorizations of AI are grounded in dualistic thinking and traditional views of technology, oversimplifying real-world settings. This article works to unfold modes of existence at play in AI/human relations. Materials from ethnographic fieldwork are used to highlight the significance of autonomy in AI/human relations. The analysis suggests that the idea of autonomy is a double-edged sword, showing that humans not only coordinate their perception of autonomy but also switch between registers by sometimes ascribing certain autonomous features to the AI system and in other situations denying the system such features. As a result, AI/human relations prove to be not so much determined by any ostensive delegation of tasks as by the way in which AI and humans engage with each other in practice. The article suggests a theory of relationality that redirects focus away from questions of agency towards questions of what it means to be in relations.

Nursing Perceptions of Robotic Technology in Healthcare: A Pretest-Posttest Survey Analysis Using an Educational Video

OCCUPATIONAL APPLICATIONSWe used a survey to evaluate the perceptions of nurses and nursing students on robotic technology for nursing care before and after reviewing an educational video that included examples of medical, care, and healthcare service robotic technology. We found that the perception of robotic technology was innately favorable and became more favorable after the video. It is beneficial for engineers to incorporate nurses' frontline knowledge into the design process from the beginning, while functional changes can be implemented since nurses comprise the largest group of healthcare professionals in hospitals and are the end users of technological devices. Educating nurses in state-of-the-art technology specific to what designers are developing can enable them to provide relevant insight. Designers and engineers can use this insight to create user-friendly, effective technology that improves not only patient care but also nurse job satisfaction.

Design and evaluation of a wearable vascular interventional surgical robot system

BACKGROUND

Remote-controlled robotic vascular interventional surgery can reduce radiation exposure to interventional physicians and improve safety. However, inconvenient operation and lack of force feedback limit its application.

MATERIALS AND METHODS

A new wearable robotic system for vascular interventional surgery is designed, which is more flexible in operation. It ensures the safety of surgery through haptic force feedback. The system was evaluated by human vascular models and animal experiments.

RESULTS

The average static error of the system is 0.048 mm when the axial motion is 250 mm and 1.259° when the rotational motion is 400°. The average error of the force feedback is 0.021 N. The results of vascular model experiments and animal experiments demonstrate the feasibility and safety of the system.

CONCLUSIONS

The proposed robotic system can assist physicians in remotely delivering standard catheters or guidewires. The system is more flexible and uses haptic force feedback to ensure surgical safety.

Clean visual field reconstruction in robot-assisted laparoscopic surgery based on dynamic prediction

Robot-assisted minimally invasive surgery has been broadly employed in complicated operations. However, the multiple surgical instruments may occupy a large amount of visual space in complex operations performed in narrow spaces, which affects the surgeon's judgment on the shape and position of the lesion as well as the course of its adjacent vessels/lacunae. In this paper, a surgical scene reconstruction method is proposed, which involves the tracking and removal of surgical instruments and the dynamic prediction of the obscured region. For tracking and segmentation of instruments, the image sequences are preprocessed by a modified U-Net architecture composed of a pre-trained ResNet101 encoder and a redesigned decoder. Also, the segmentation boundaries of the instrument shafts are extended using image filtering and a real-time index mask algorithm to achieve precise localization of the obscured elements. For predicting the deformation of soft tissues, a soft tissue deformation prediction algorithm is proposed based on dense optical flow gravitational field and entropy increase, which can achieve local dynamic visualization of the surgical scene by integrating image morphological operations. Finally, the preliminary experiments and the pre-clinical evaluation were presented to demonstrate the performance of the proposed method. The results show that the proposed method can provide the surgeon with a clean and comprehensive surgical scene, reconstruct the course of important vessels/lacunae, and avoid inadvertent injuries.

Robotic ultrasound imaging: State-of-the-art and future perspectives

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.

Preoperative Planning Framework for Robot-Assisted Dental Implant Surgery: Finite-Parameter Surrogate Model and Optimization of Instrument Placement

For robot-assisted dental implant surgery, it is necessary to feed the instrument into a specified position to perform surgery. To improve safety and efficiency, a preoperative planning framework, including a finite-parameter surrogate model (FPSM) and an automatic instrument-placement method, is proposed in this paper. This framework is implemented via two-stage optimization. In the first stage, a group of closed curves in polar coordinates is used to represent the oral cavity. By optimizing a finite number of parameters for these curves, the oral structure is simplified to form the FPSM. In the second stage, the FPSM serves as a fast safety estimator with which the target position/orientation of the instrument for the feeding motion is automatically determined through particle swarm optimization (PSO). The optimized feeding target can be used to generate a virtual fixture (VF) to avoid undesired operations and to lower the risk of collision. This proposed framework has the advantages of being safe, fast, and accurate, overcoming the computational burden and insufficient real-time performance of complex 3D models. The framework has been developed and tested, preliminarily verifying its feasibility, efficiency, and effectiveness.

A collaborative robotic platform for sensor-aware fibula osteotomies in mandibular reconstruction surgery

Precision and safety are crucial in performing fibula osteotomy during mandibular reconstruction with free fibula flap (FFF). However, current clinical methods, such as template-guided osteotomy, have the potential to cause damage to fibular vessels. To address the challenge, this paper introduces the development of the surgical robot for fibula osteotomies in mandibular reconstruction surgery and propose an algorithm for sensor-aware hybrid force-motion control for safe osteotomy, which includes three parts: osteotomy motion modeling from surgeons' demonstrations, Dynamic-system-based admittance control and osteotomy sawed-through detection. As a result, the average linear variation of the osteotomized segments was 1.08±0.41mm, and the average angular variation was 1.32±0.65∘. The threshold of osteotomy sawed-through detection is 0.5 at which the average offset is 0.5mm. In conclusion, with the assistance of surgical robot for mandibular reconstruction, surgeons can perform fibula osteotomy precisely and safely.

Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future.

Flexible robotic endoscopy for treating gastrointestinal neoplasms

Therapeutic flexible endoscopic robotic systems have been developed primarily as a platform for endoscopic submucosal dissection (ESD) in the treatment of early-stage gastrointestinal cancer. Since ESD can only be performed by highly skilled endoscopists, the goal is to lower the technical hurdles to ESD by introducing a robot. In some cases, such robots have already been used clinically, but they are still in the research and development stage. This paper outlined the current status of development, including a system by the author’s group, and discussed future challenges.

Designing robots that do no harm: understanding the challenges of Ethics for Robots

This article describes key challenges in creating an ethics "for" robots. Robot ethics is not only a matter of the effects caused by robotic systems or the uses to which they may be put, but also the ethical rules and principles that these systems ought to follow-what we call "Ethics for Robots." We suggest that the Principle of Nonmaleficence, or "do no harm," is one of the basic elements of an ethics for robots-especially robots that will be used in a healthcare setting. We argue, however, that the implementation of even this basic principle will raise significant challenges for robot designers. In addition to technical challenges, such as ensuring that robots are able to detect salient harms and dangers in the environment, designers will need to determine an appropriate sphere of responsibility for robots and to specify which of various types of harms must be avoided or prevented. These challenges are amplified by the fact that the robots we are currently able to design possess a form of semi-autonomy that differs from other more familiar semi-autonomous agents such as animals or young children. In short, robot designers must identify and overcome the key challenges of an ethics for robots before they may ethically utilize robots in practice.

Artificial intelligence and automation in endoscopy and surgery

Modern endoscopy relies on digital technology, from high-resolution imaging sensors and displays to electronics connecting configurable illumination and actuation systems for robotic articulation. In addition to enabling more effective diagnostic and therapeutic interventions, the digitization of the procedural toolset enables video data capture of the internal human anatomy at unprecedented levels. Interventional video data encapsulate functional and structural information about a patient's anatomy as well as events, activity and action logs about the surgical process. This detailed but difficult-to-interpret record from endoscopic procedures can be linked to preoperative and postoperative records or patient imaging information. Rapid advances in artificial intelligence, especially in supervised deep learning, can utilize data from endoscopic procedures to develop systems for assisting procedures leading to computer-assisted interventions that can enable better navigation during procedures, automation of image interpretation and robotically assisted tool manipulation. In this Perspective, we summarize state-of-the-art artificial intelligence for computer-assisted interventions in gastroenterology and surgery.

3D Autonomous Surgeon's Hand Movement Assessment Using a Cascaded Fuzzy Supervisor in Multi-Thread Video Processing

The purpose of the Fundamentals of Laparoscopic Surgery (FLS) training is to develop laparoscopic surgery skills by using simulation experiences. Several advanced training methods based on simulation have been created to enable training in a non-patient environment. Laparoscopic box trainers-cheap, portable devices-have been deployed for a while to offer training opportunities, competence evaluations, and performance reviews. However, the trainees must be under the supervision of medical experts who can evaluate their abilities, which is an expensive and time-consuming operation. Thus, a high level of surgical skill, determined by assessment, is necessary to prevent any intraoperative issues and malfunctions during a real laparoscopic procedure and during human intervention. To guarantee that the use of laparoscopic surgical training methods results in surgical skill improvement, it is necessary to measure and assess surgeons' skills during tests. We used our intelligent box-trainer system (IBTS) as a platform for skill training. The main aim of this study was to monitor the surgeon's hands' movement within a predefined field of interest. To evaluate the surgeons' hands' movement in 3D space, an autonomous evaluation system using two cameras and multi-thread video processing is proposed. This method works by detecting laparoscopic instruments and using a cascaded fuzzy logic assessment system. It is composed of two fuzzy logic systems executing in parallel. The first level assesses the left and right-hand movements simultaneously. Its outputs are cascaded by the final fuzzy logic assessment at the second level. This algorithm is completely autonomous and removes the need for any human monitoring or intervention. The experimental work included nine physicians (surgeons and residents) from the surgery and obstetrics/gynecology (OB/GYN) residency programs at WMU Homer Stryker MD School of Medicine (WMed) with different levels of laparoscopic skills and experience. They were recruited to participate in the peg-transfer task. The participants' performances were assessed, and the videos were recorded throughout the exercises. The results were delivered autonomously about 10 s after the experiments were concluded. In the future, we plan to increase the computing power of the IBTS to achieve real-time performance assessment.

Integration of Reinforcement Learning in a Virtual Robotic Surgical Simulation

Background. The revolutions in AI hold tremendous capacity to augment human achievements in surgery, but robust integration of deep learning algorithms with high-fidelity surgical simulation remains a challenge. We present a novel application of reinforcement learning (RL) for automating surgical maneuvers in a graphical simulation.Methods. In the Unity3D game engine, the Machine Learning-Agents package was integrated with the NVIDIA FleX particle simulator for developing autonomously behaving RL-trained scissors. Proximal Policy Optimization (PPO) was used to reward movements and desired behavior such as movement along desired trajectory and optimized cutting maneuvers along the deformable tissue-like object. Constant and proportional reward functions were tested, and TensorFlow analytics was used to informed hyperparameter tuning and evaluate performance.Results. RL-trained scissors reliably manipulated the rendered tissue that was simulated with soft-tissue properties. A desirable trajectory of the autonomously behaving scissors was achieved along 1 axis. Proportional rewards performed better compared to constant rewards. Cumulative reward and PPO metrics did not consistently improve across RL-trained scissors in the setting for movement across 2 axes (horizontal and depth).Conclusion. Game engines hold promising potential for the design and implementation of RL-based solutions to simulated surgical subtasks. Task completion was sufficiently achieved in one-dimensional movement in simulations with and without tissue-rendering. Further work is needed to optimize network architecture and parameter tuning for increasing complexity.

Inducing Performance of Commercial Surgical Robots in Space

Pre-existing surgical robotic systems are sold with electronics (sensors and controllers) that can prove difficult to retroactively improve when newly developed methods are proposed. Improvements must be somehow "imposed" upon the original robotic systems. What options are available for imposing performance from pre-existing, common systems and how do the options compare? Optimization often assumes idealized systems leading to open-loop results (lacking feedback from sensors), and this manuscript investigates utility of prefiltering, such other modern methods applied to non-idealized systems, including fusion of noisy sensors and so-called "fictional forces" associated with measurement of displacements in rotating reference frames. A dozen modern approaches are compared as the main contribution of this work. Four methods are idealized cases establishing a valid theoretical comparative benchmark. Subsequently, eight modern methods are compared against the theoretical benchmark and against the pre-existing robotic systems. The two best performing methods included one modern application of a classical approach (velocity control) and one modern approach derived using Pontryagin's methods of systems theory, including Hamiltonian minimization, adjoint equations, and terminal transversality of the endpoint Lagrangian. The key novelty presented is the best performing method called prefiltered open-loop optimal + transport decoupling, achieving 1-3 percent attitude tracking performance of the robotic instrument with a two percent reduced computational burden and without increased costs (effort).

Design and evaluation of vascular interventional robot system for complex coronary artery lesions

At present, most vascular intervention robots cannot cope with the more common coronary complex lesions in the clinic. Moreover, the lack of effective force feedback increases the risk of surgery. In this paper, a vascular interventional robot that can collaboratively deliver multiple interventional instruments has been developed to assist doctors in the operation of complex lesions. Based on the doctor's skills and the delivery principle of interventional instruments, the main and slave manipulators of the robot system are designed. Haptic force feedback is generated through resistance measuring mechanism and active drag system. In addition, a force feedback control strategy based on force-velocity mapping is proposed to realize the continuous change of force and avoid vibration. The proposed robot system was evaluated through a series of experiments. The experimental results show that the system can accurately measure the delivery resistance of interventional instruments, and provide haptic force feedback to doctors. The capability of the system to collaboratively deliver multiple interventional instruments is effective. Therefore, it can be considered that the robot system is feasible and effective.

Robot-Assisted Rehabilitation Architecture Supported by a Distributed Data Acquisition System

Rehabilitation robotics aims to facilitate the rehabilitation procedure for patients and physical therapists. This field has a relatively long history dating back to the 1990s; however, their implementation and the standardisation of their application in the medical field does not follow the same pace, mainly due to their complexity of reproduction and the need for their approval by the authorities. This paper aims to describe architecture that can be applied to industrial robots and promote their application in healthcare ecosystems. The control of the robotic arm is performed using the software called SmartHealth, offering a 2 Degree of Autonomy (DOA). Data are gathered through electromyography (EMG) and force sensors at a frequency of 45 Hz. It also proves the capabilities of such small robots in performing such medical procedures. Four exercises focused on shoulder rehabilitation (passive, restricted active-assisted, free active-assisted and Activities of Daily Living (ADL)) were carried out and confirmed the viability of the proposed architecture and the potential of small robots (i.e., the UR3) in rehabilitation procedure accomplishment. This robot can perform the majority of the default exercises in addition to ADLs but, nevertheless, their limits were also uncovered, mainly due to their limited Range of Motion (ROM) and cost.

Characterization of the Workspace and Limits of Operation of Laser Treatments for Vascular Lesions of the Lower Limbs

The increase of the aging population brings numerous challenges to health and aesthetic segments. Here, the use of laser therapy for dermatology is expected to increase since it allows for non-invasive and infection-free treatments. However, existing laser devices require doctors' manually handling and visually inspecting the skin. As such, the treatment outcome is dependent on the user's expertise, which frequently results in ineffective treatments and side effects. This study aims to determine the workspace and limits of operation of laser treatments for vascular lesions of the lower limbs. The results of this study can be used to develop a robotic-guided technology to help address the aforementioned problems. Specifically, workspace and limits of operation were studied in eight vascular laser treatments. For it, an electromagnetic tracking system was used to collect the real-time positioning of the laser during the treatments. The computed average workspace length, height, and width were 0.84 ± 0.15, 0.41 ± 0.06, and 0.78 ± 0.16 m, respectively. This corresponds to an average volume of treatment of 0.277 ± 0.093 m³. The average treatment time was 23.2 ± 10.2 min, with an average laser orientation of 40.6 ± 5.6 degrees. Additionally, the average velocities of 0.124 ± 0.103 m/s and 31.5 + 25.4 deg/s were measured. This knowledge characterizes the vascular laser treatment workspace and limits of operation, which may ease the understanding for future robotic system development.

COVID-19 and public support for autonomous technologies—Did the pandemic catalyze a world of robots?

By introducing a novel risk to human interaction, COVID-19 may have galvanized interest in uses of artificial intelligence (AI). But was the pandemic a large enough catalyst to change public attitudes about the costs and benefits of autonomous systems whose operations increasingly rely on AI? To answer this question, we use a preregistered research design that exploits variation across the 2018 and 2020 waves of the CCES/CES, a nationally representative survey of adults in the United States. We compare support for autonomous cars, autonomous surgeries, weapons, and cyber defense pre- and post-the beginning of the COVID-19 pandemic. We find that, despite the incentives created by COVID-19, the pandemic did not increase support for most of these technologies, except in the case of autonomous surgery among those who know someone who died of COVID-19. The results hold even when controlling for a variety of relevant political and demographic factors. The pandemic did little to push potential autonomous vehicle users to support adoption. Further, American concerns about autonomous weapons, including cyber defense, remain sticky and perhaps exacerbated over the last two years. These findings suggest that the relationship between the COVID-19 pandemic and the adoption of many of these systems is far more nuanced and complex than headlines may suggest.

An Overview of the Current State and Perspectives of Pharmacy Robot and Medication Dispensing Technology

It has been widely reported that a large number of patients die from cases of errors in the issuing of medication prescriptions. These cases occur due to a wide range of things, but the common denominator in all of the cases is humans. A hospital pharmacy has a very critical task, especially with growing patient numbers. The increasing number of prescriptions needed to be filled daily reduces the amount of time that the staff can use to focus on each individual prescription, which may increase the human error ratio. The need for robotic-assisted pharmacies is arising from here to distribute drugs to eradicate or substantially reduce human error. The pharmacy robot is one of the most significant technologies that play a prominent role in the advancement of hospital pharmacy systems. The purpose of this review paper is to cover the pharmacy robot concept and the published literature reporting on pharmacy robot technology as one of the most important applications of artificial intelligence (AI) in pharmacology. Although the outcomes of the impact of the pharmacy robot have been increasingly beneficial in overall improvement, staff morale, and functionality of pharmacies, there are still mechanical errors occurring. The errors, in turn, require human intervention. The key takeaway from this study is that robots or machines cannot replace human duties in their entirety. This in turn means that those human interventions will have an impact on the workflow and throughput.

Sensor-Based Automated Detection of Electrosurgical Cautery States

In computer-assisted surgery, it is typically required to detect when the tool comes into contact with the patient. In activated electrosurgery, this is known as the energy event. By continuously tracking the electrosurgical tools' location using a navigation system, energy events can help determine locations of sensor-classified tissues. Our objective was to detect the energy event and determine the settings of electrosurgical cautery-robustly and automatically based on sensor data. This study aims to demonstrate the feasibility of using the cautery state to detect surgical incisions, without disrupting the surgical workflow. We detected current changes in the wires of the cautery device and grounding pad using non-invasive current sensors and an oscilloscope. An open-source software was implemented to apply machine learning on sensor data to detect energy events and cautery settings. Our methods classified each cautery state at an average accuracy of 95.56% across different tissue types and energy level parameters altered by surgeons during an operation. Our results demonstrate the feasibility of automatically identifying energy events during surgical incisions, which could be an important safety feature in robotic and computer-integrated surgery. This study provides a key step towards locating tissue classifications during breast cancer operations and reducing the rate of positive margins.

Force-based Control for Safe Robot-assisted Retinal Interventions: In Vivo Evaluation in Animal Studies

In recent years, robotic assistance in vitreoretinal surgery has moved from a benchtop environment to the operating rooms. Emerging robotic systems improve tool manoeuvrability and provide precise tool motions in a constrained intraocular environment and reduce/remove hand tremor. However, often due to their stiff and bulky mechanical structure, they diminish the perception of tool-to-sclera (scleral) forces, on which the surgeon relies, for eyeball manipulation. In this paper we measure these scleral forces and actively control the robot to keep them under a predefined threshold. Scleral forces are measured using a Fiber Bragg Grating (FBG) based force sensing instrument in an in vivo rabbit eye model in manual, cooperative robotic assistance with no scleral force control (NC), adaptive scleral force norm control (ANC) and adaptive scleral force component control (ACC) methods. To the best of our knowledge, this is the first time that the scleral forces are measured in an in vivo eye model during robot assisted vitreoretinal procedures. An experienced retinal surgeon repeated an intraocular tool manipulation (ITM) task 10 times in four in vivo rabbit eyes and a phantom eyeball, for a total of 50 repetitions in each control mode. Statistical analysis shows that the ANC and ACC control schemes restrict the duration of the undesired scleral forces to 4.41% and 14.53% as compared to 43.30% and 35.28% in manual and NC cases, respectively during the in vivo studies. These results show that the active robot control schemes can maintain applied scleral forces below a desired threshold during robot-assisted vitreoretinal surgery. The scleral forces measurements in this study may enable a better understanding of tool-to-sclera interactions during vitreoretinal surgery and the proposed control strategies could be extended to other microsurgery and robot-assisted interventions.

Concepts and Trends n Autonomy for Robot-Assisted Surgery

Surgical robots have been widely adopted with over 4000 robots being used in practice daily. However, these are telerobots that are fully controlled by skilled human surgeons. Introducing "surgeon-assist"-some forms of autonomy-has the potential to reduce tedium and increase consistency, analogous to driver-assist functions for lanekeeping, cruise control, and parking. This article examines the scientific and technical backgrounds of robotic autonomy in surgery and some ethical, social, and legal implications. We describe several autonomous surgical tasks that have been automated in laboratory settings, and research concepts and trends.

Monopolar, bipolar, tripolar, and tetrapolar configurations in robot assisted electrical impedance scanning

OBJECTIVE

Tissue recognition is a critical process during a Robot-assisted minimally invasive surgery (RMIS) and it relies on the involvement of advanced sensing technology.

APPROACH

In this paper, the concept of Robot Assisted Electrical Impedance Sensing (RAEIS) is utilized and further developed aiming to sense the electrical bioimpedance of target tissue directly based on the existing robotic instruments and control strategy. Specifically, we present a new sensing configuration called pseudo-tetrapolar method. With the help of robotic control, we can achieve a similar configuration as traditional tetrapolar, and with better accuracy.

MAIN RESULTS

Five configurations including monopolar, bipolar, tripolar, tetrapolar and pseudo-tetrapolar are analyzed and compared through simulation experiments. Advantages and disadvantages of each configuration are thus discussed.

SIGNIFICANCE

This study investigates the measurement of tissue electrical property directly based on the existing robotic surgical instruments. Specifically, different sensing configurations can be realized through different connection and control strategies, making them suitable for different application scenarios.

Infrastructural Requirements and Regulatory Challenges of a Sustainable Urban Air Mobility Ecosystem

The United Nations has long put on the discussion agenda the sustainability challenges of urbanization, which have both direct and indirect effects on future regulation strategies. Undoubtedly, most initiatives target better quality of life, improved access to services & goods and environment protection. As commercial aerial urban transportation may become a feasible research goal in the near future, the connection possibilities between cities and regions scale up. It is expected that the growing number of vertical takeoff & landing vehicles used for passenger and goods transportation will change the infrastructure of the cities, and will have a significant effect on the cityscapes as well. In addition to the widely discussed regulatory and safety issues, the introduction of elevated traffic also raises environmental concerns, which influences the existing and required service and control infrastructure, and thus significantly affects sustainability. This paper provides narrated overview of the most common aspects of safety, licensing and regulations for passenger vertical takeoff & landing vehicles, and highlights the most important aspects of infrastructure planning, design and operation, which should be taken into account to maintain and efficiently operate this new way of transportation, leading to a sustainable urban air mobility ecosystem.

Fuzzy logic supervisor –A surgical skills assessment system using multi-class detection of laparoscopic box-trainer instruments

Recent developments in deep learning can be used in skill assessments for laparoscopic surgeons. In Minimally Invasive Surgery (MIS), surgeons should acquire many skills before carrying out a real operation. The Laparoscopic Surgical Box-Trainer allows surgery residents to train on specific skills that are not traditionally taught to them. This study aims to automatically detect the tips of laparoscopic instruments, localize a point, evaluate the detection accuracy to provide valuable assessment and expedite the development of surgery skills and assess the trainees’ performance using a Multi-Input-Single-Output Fuzzy Logic Supervisor system. The output of the fuzzy logic assessment is the performance evaluation for the surgeon, and it is quantified in percentages. Based on the experimental results, the trained SSD Mobilenet V2 FPN can identify each instrument at a score of 70% fidelity. On the other hand, the trained SSD ResNet50 V1 FPN can detect each instrument at the score of 90% fidelity, in each location within a region of interest, and determine their relative distance with over 65% and 80% reliability, respectively. This method can be applied in different types of laparoscopic tooltip detection. Because there were a few instances when the detection failed, and the system was designed to generate pass-fail assessment, we recommend improving the measurement algorithm and the performance assessment by adding a camera to the system and measuring the distance from multiple perspectives.

Snake Robots for Surgical Applications: A Review

Although substantial advancements have been achieved in robot-assisted surgery, the blueprint to existing snake robotics predominantly focuses on the preliminary structural design, control, and human–robot interfaces, with features which have not been particularly explored in the literature. This paper aims to conduct a review of planning and operation concepts of hyper-redundant serpentine robots for surgical use, as well as any future challenges and solutions for better manipulation. Current researchers in the field of the manufacture and navigation of snake robots have faced issues, such as a low dexterity of the end-effectors around delicate organs, state estimation and the lack of depth perception on two-dimensional screens. A wide range of robots have been analysed, such as the i²Snake robot, inspiring the use of force and position feedback, visual servoing and augmented reality (AR). We present the types of actuation methods, robot kinematics, dynamics, sensing, and prospects of AR integration in snake robots, whilst addressing their shortcomings to facilitate the surgeon’s task. For a smoother gait control, validation and optimization algorithms such as deep learning databases are examined to mitigate redundancy in module linkage backlash and accidental self-collision. In essence, we aim to provide an outlook on robot configurations during motion by enhancing their material compositions within anatomical biocompatibility standards.

Performance and Capability Assessment in Surgical Subtask Automation

Robot-Assisted Minimally Invasive Surgery (RAMIS) has reshaped the standard clinical practice during the past two decades. Many believe that the next big step in the advancement of RAMIS will be partial autonomy, which may reduce the fatigue and the cognitive load on the surgeon by performing the monotonous, time-consuming subtasks of the surgical procedure autonomously. Although serious research efforts are paid to this area worldwide, standard evaluation methods, metrics, or benchmarking techniques are still not formed. This article aims to fill the void in the research domain of surgical subtask automation by proposing standard methodologies for performance evaluation. For that purpose, a novel characterization model is presented for surgical automation. The current metrics for performance evaluation and comparison are overviewed and analyzed, and a workflow model is presented that can help researchers to identify and apply their choice of metrics. Existing systems and setups that serve or could serve as benchmarks are also introduced and the need for standard benchmarks in the field is articulated. Finally, the matter of Human-Machine Interface (HMI) quality, robustness, and the related legal and ethical issues are presented.

PlatypOUs-A Mobile Robot Platform and Demonstration Tool Supporting STEM Education

Given the rising popularity of robotics, student-driven robot development projects are playing a key role in attracting more people towards engineering and science studies. This article presents the early development process of an open-source mobile robot platform-named PlatypOUs-which can be remotely controlled via an electromyography (EMG) appliance using the MindRove brain-computer interface (BCI) headset as a sensor for the purpose of signal acquisition. The gathered bio-signals are classified by a Support Vector Machine (SVM) whose results are translated into motion commands for the mobile platform. Along with the physical mobile robot platform, a virtual environment was implemented using Gazebo (an open-source 3D robotic simulator) inside the Robot Operating System (ROS) framework, which has the same capabilities as the real-world device. This can be used for development and test purposes. The main goal of the PlatypOUs project is to create a tool for STEM education and extracurricular activities, particularly laboratory practices and demonstrations. With the physical robot, the aim is to improve awareness of STEM outside and beyond the scope of regular education programmes. It implies several disciplines, including system design, control engineering, mobile robotics and machine learning with several application aspects in each. Using the PlatypOUs platform and the simulator provides students and self-learners with a firsthand exercise, and teaches them to deal with complex engineering problems in a professional, yet intriguing way.

Autonomous robotic laparoscopic surgery for intestinal anastomosis

Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeon's skill and experience. Autonomous anastomosis is a challenging soft-tissue surgery task because it requires intricate imaging, tissue tracking, and surgical planning techniques, as well as a precise execution via highly adaptable control strategies often in unstructured and deformable environments. In the laparoscopic setting, such surgeries are even more challenging because of the need for high maneuverability and repeatability under motion and vision constraints. Here we describe an enhanced autonomous strategy for laparoscopic soft tissue surgery and demonstrate robotic laparoscopic small bowel anastomosis in phantom and in vivo intestinal tissues. This enhanced autonomous strategy allows the operator to select among autonomously generated surgical plans and the robot executes a wide range of tasks independently. We then use our enhanced autonomous strategy to perform in vivo autonomous robotic laparoscopic surgery for intestinal anastomosis on porcine models over a 1-week survival period. We compared the anastomosis quality criteria-including needle placement corrections, suture spacing, suture bite size, completion time, lumen patency, and leak pressure-of the developed autonomous system, manual laparoscopic surgery, and robot-assisted surgery (RAS). Data from a phantom model indicate that our system outperforms expert surgeons' manual technique and RAS technique in terms of consistency and accuracy. This was also replicated in the in vivo model. These results demonstrate that surgical robots exhibiting high levels of autonomy have the potential to improve consistency, patient outcomes, and access to a standard surgical technique.

Robot-assisted surgery in space: pros and cons. A review from the surgeon's point of view

The target of human flight in space has changed from permanence on the International Space Station to missions beyond low earth orbit and the Lunar Gateway for deep space exploration and Missions to Mars. Several conditions affecting space missions had to be considered: for example the effect of weightlessness and radiations on the human body, behavioral health decrements or communication latency, and consumable resupply. Telemedicine and telerobotic applications, robot-assisted surgery with some hints on experimental surgical procedures carried out in previous missions, had to be considered as well. The need for greater crew autonomy in health issues is related to the increasing severity of medical and surgical interventions that could occur in these missions, and the presence of a highly trained surgeon on board would be recommended. A surgical robot could be a valuable aid but only inasfar as it is provided with multiple functions, including the capability to perform certain procedures autonomously. Space missions in deep space or on other planets present new challenges for crew health. Providing a multi-function surgical robot is the new frontier. Research in this field shall be paving the way for the development of new structured plans for human health in space, as well as providing new suggestions for clinical applications on Earth.

Shape Reconstruction Processes for Interventional Application Devices: State of the Art, Progress, and Future Directions

Soft and continuum robots are transforming medical interventions thanks to their flexibility, miniaturization, and multidirectional movement abilities. Although flexibility enables reaching targets in unstructured and dynamic environments, it also creates challenges for control, especially due to interactions with the anatomy. Thus, in recent years lots of efforts have been devoted for the development of shape reconstruction methods, with the advancement of different kinematic models, sensors, and imaging techniques. These methods can increase the performance of the control action as well as provide the tip position of robotic manipulators relative to the anatomy. Each method, however, has its advantages and disadvantages and can be worthwhile in different situations. For example, electromagnetic (EM) and Fiber Bragg Grating (FBG) sensor-based shape reconstruction methods can be used in small-scale robots due to their advantages thanks to miniaturization, fast response, and high sensitivity. Yet, the problem of electromagnetic interference in the case of EM sensors, and poor response to high strains in the case of FBG sensors need to be considered. To help the reader make a suitable choice, this paper presents a review of recent progress on shape reconstruction methods, based on a systematic literature search, excluding pure kinematic models. Methods are classified into two categories. First, sensor-based techniques are presented that discuss the use of various sensors such as FBG, EM, and passive stretchable sensors for reconstructing the shape of the robots. Second, imaging-based methods are discussed that utilize images from different imaging systems such as fluoroscopy, endoscopy cameras, and ultrasound for the shape reconstruction process. The applicability, benefits, and limitations of each method are discussed. Finally, the paper draws some future promising directions for the enhancement of the shape reconstruction methods by discussing open questions and alternative methods.

A Novel Perception Framework for Automatic Laparoscope Zoom Factor Control Using Tool Geometry

In conventional Minimally Invasive Surgery, the surgeon conducts the operation while a human or robot holds the laparoscope. Laparoscope control is returned to the surgeon in teleoperated camera holding robots, but simultaneously controlling the laparoscope and surgical tools might be cognitively demanding. On the other hand, fully automated camera holders are still limited in their performance. To help the surgeon to better focus on the main operation while maintaining their control authority, we propose an automatic laparoscope zoom factor control framework for Robot-Assisted Minimally Invasive Surgery. In this paper, we present the perception section of the framework. It extracts and uses the surgical tool's geometric characteristics to adjust the laparoscope's zoom factor, without any artificial markers. The acceptable range and tooltip's position frequency during operations are analysed based on the gallbladder removal surgery dataset (Cholec80). The common range and tooltip's heatmap are identified and presented quantitatively.

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

Minimally invasive robotic surgery copes with some disadvantages for the surgeon of minimally invasive surgery while preserving the advantages for the patient. Most commercially available robotic systems are telemanipulated with haptic input devices. The exploitation of the haptics channel, e.g., by means of Virtual Fixtures, would allow for an individualized enhancement of surgical performance with contextual assistance. However, it remains an open field of research as it is non-trivial to estimate the task context itself during a surgery. In contrast, surgical training allows to abstract away from a real operation and thus makes it possible to model the task accurately. The presented approach exploits this fact to parameterize Virtual Fixtures during surgical training, proposing a Shared Control Parametrization Engine that retrieves procedural context information from a Digital Twin. This approach accelerates a proficient use of the robotic system for novice surgeons by augmenting the surgeon's performance through haptic assistance. With this our aim is to reduce the required skill level and cognitive load of a surgeon performing minimally invasive robotic surgery. A pilot study is performed on the DLR MiroSurge system to evaluate the presented approach. The participants are tasked with two benchmark scenarios of surgical training. The execution of the benchmark scenarios requires basic skills as pick, place and path following. The evaluation of the pilot study shows the promising trend that novel users profit from the haptic augmentation during training of certain tasks.

Uncertainty-Aware Knowledge Distillation for Collision Identification of Collaborative Robots

Human-robot interaction has received a lot of attention as collaborative robots became widely utilized in many industrial fields. Among techniques for human-robot interaction, collision identification is an indispensable element in collaborative robots to prevent fatal accidents. This paper proposes a deep learning method for identifying external collisions in 6-DoF articulated robots. The proposed method expands the idea of CollisionNet, which was previously proposed for collision detection, to identify the locations of external forces. The key contribution of this paper is uncertainty-aware knowledge distillation for improving the accuracy of a deep neural network. Sample-level uncertainties are estimated from a teacher network, and larger penalties are imposed for uncertain samples during the training of a student network. Experiments demonstrate that the proposed method is effective for improving the performance of collision identification.

Endoscopic Image-Based Skill Assessment in Robot-Assisted Minimally Invasive Surgery

Objective skill assessment-based personal performance feedback is a vital part of surgical training. Either kinematic-acquired through surgical robotic systems, mounted sensors on tooltips or wearable sensors-or visual input data can be employed to perform objective algorithm-driven skill assessment. Kinematic data have been successfully linked with the expertise of surgeons performing Robot-Assisted Minimally Invasive Surgery (RAMIS) procedures, but for traditional, manual Minimally Invasive Surgery (MIS), they are not readily available as a method. 3D visual features-based evaluation methods tend to outperform 2D methods, but their utility is limited and not suited to MIS training, therefore our proposed solution relies on 2D features. The application of additional sensors potentially enhances the performance of either approach. This paper introduces a general 2D image-based solution that enables the creation and application of surgical skill assessment in any training environment. The 2D features were processed using the feature extraction techniques of a previously published benchmark to assess the attainable accuracy. We relied on the JHU-ISI Gesture and Skill Assessment Working Set dataset-co-developed by the Johns Hopkins University and Intuitive Surgical Inc. Using this well-established set gives us the opportunity to comparatively evaluate different feature extraction techniques. The algorithm reached up to 95.74% accuracy in individual trials. The highest mean accuracy-averaged over five cross-validation trials-for the surgical subtask of Knot-Tying was 83.54%, for Needle-Passing 84.23% and for Suturing 81.58%. The proposed method measured well against the state of the art in 2D visual-based skill assessment, with more than 80% accuracy for all three surgical subtasks available in JIGSAWS (Knot-Tying, Suturing and Needle-Passing). By introducing new visual features-such as image-based orientation and image-based collision detection-or, from the evaluation side, utilising other Support Vector Machine kernel methods, tuning the hyperparameters or using other classification methods (e.g., the boosted trees algorithm) instead, classification accuracy can be further improved. We showed the potential use of optical flow as an input for RAMIS skill assessment, highlighting the maximum accuracy achievable with these data by evaluating it with an established skill assessment benchmark, by evaluating its methods independently. The highest performing method, the Residual Neural Network, reached means of 81.89%, 84.23% and 83.54% accuracy for the skills of Suturing, Needle-Passing and Knot-Tying, respectively.

Force-guided autonomous robotic ultrasound scanning control method for soft uncertain environment

PURPOSE

Autonomous ultrasound imaging by robotic ultrasound scanning systems in complex soft uncertain clinical environments is important and challenging to assist in therapy. To cope with the complex environment faced by the ultrasound probe during the scanning process, we propose an autonomous robotic ultrasound (US) control method based on reinforcement learning (RL) model to build the relationship between the environment and the system. The proposed method requires only contact force as input information to achieve robot control of the posture and contact force of the probe without any a priori information about the target and the environment.

METHODS

First, an RL agent is proposed and trained by a policy gradient theorem-based RL model with the 6-degree-of-freedom (DOF) contact force of the US probe to learn the relationship between contact force and output force directly. Then, a force control strategy based on the admittance controller is proposed for synchronous force, orientation and position control by defining the desired contact force as the action space.

RESULTS

The proposed method was evaluated via collected US images, contact force and scan trajectories by scanning an unknown soft phantom. The experimental results indicated that the proposed method differs from the free-hand scanned approach in the US images within 3 ± 0.4%. The analysis results of contact forces and trajectories indicated that our method could make stable scanning processes on a soft uncertain skin surface and obtained US images.

CONCLUSION

We propose a concise and efficient force-guided US robot scanning control method for soft uncertain environment based on reinforcement learning. Experimental results validated our method's feasibility and validity for complex skin surface scanning, and the volunteer experiments indicated the potential application value in the complex clinical environment of robotic US imaging system especially with limited visual information.