Real-time surgical tool detection with multi-scale positional encoding and contrastive learning

Real-time detection of surgical tools in laparoscopic data plays a vital role in understanding surgical procedures, evaluating the performance of trainees, facilitating learning, and ultimately supporting the autonomy of robotic systems. Existing detection methods for surgical data need to improve processing speed and high prediction accuracy. Most methods rely on anchors or region proposals, limiting their adaptability to variations in tool appearance and leading to sub-optimal detection results. Moreover, using non-anchor-based detectors to alleviate this problem has been partially explored without remarkable results. An anchor-free architecture based on a transformer that allows real-time tool detection is introduced. The proposal is to utilize multi-scale features within the feature extraction layer and at the transformer-based detection architecture through positional encoding that can refine and capture context-aware and structural information of different-sized tools. Furthermore, a supervised contrastive loss is introduced to optimize representations of object embeddings, resulting in improved feed-forward network performances for classifying localized bounding boxes. The strategy demonstrates superiority to state-of-the-art (SOTA) methods. Compared to the most accurate existing SOTA (DSSS) method, the approach has an improvement of nearly 4% on mAP and a reduction in the inference time by 113%. It also showed a 7% higher mAP than the baseline model.

Robust endoscopic image mosaicking via fusion of multimodal estimation

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics.

Patient-Specific Magnetic Catheters for Atraumatic Autonomous Endoscopy

Despite increasing interest in minimally invasive surgical techniques and related developments in flexible endoscopes and catheters, follow-the-leader motion remains elusive. Following the path of least resistance through a tortuous and potentially delicate environment without relying on interaction with the surrounding anatomy requires the control of many degrees of freedom. This typically results in large-diameter instruments. One viable solution to obtain dexterity without increasing size is via multiple-point magnetic actuation over the length of the catheter. The main challenge of this approach is planning magnetic interaction to allow the catheter to adapt to the surrounding anatomy during navigation. We design and manufacture a fully shape-forming, soft magnetic catheter of 80 mm length and 2 mm diameter, capable of navigating a human anatomy in a follow-the-leader fashion. Although this system could be exploited for a range of endoscopic or intravascular applications, here we demonstrate its efficacy for navigational bronchoscopy. From a patient-specific preoperative scan, we optimize the catheters' magnetization profiles and the shape-forming actuating field. To generate the required transient magnetic fields, a dual-robot arm system is employed. We fabricate three separate prototypes to demonstrate minimal contact navigation through a three-dimensional bronchial tree phantom under precomputed robotic control. We also compare a further four separate optimally designed catheters against mechanically equivalent designs with axial magnetization profiles along their length and only at the tip. Using our follow-the-leader approach, we demonstrate up to 50% more accurate tracking, 50% reduction in obstacle contact time during navigation over the state of the art, and an improvement in targeting error of 90%.

Optimization and fabrication of programmable domains for soft magnetic robots: A review

Driven by the aim of realizing functional robotic systems at the milli- and submillimetre scale for biomedical applications, the area of magnetically driven soft devices has received significant recent attention. This has resulted in a new generation of magnetically controlled soft robots with patterns of embedded, programmable domains throughout their structures. This type of programmable magnetic profiling equips magnetic soft robots with shape programmable memory and can be achieved through the distribution of discrete domains (voxels) with variable magnetic densities and magnetization directions. This approach has produced highly compliant, and often bio-inspired structures that are well suited to biomedical applications at small scales, including microfluidic transport and shape-forming surgical catheters. However, to unlock the full potential of magnetic soft robots with improved designs and control, significant challenges remain in their compositional optimization and fabrication. This review considers recent advances and challenges in the interlinked optimization and fabrication aspects of programmable domains within magnetic soft robots. Through a combination of improvements in the computational capacity of novel optimization methods with advances in the resolution, material selection and automation of existing and novel fabrication methods, significant further developments in programmable magnetic soft robots may be realized.

Micro-scale aerosol jet printing of superparamagnetic Fe3O4 nanoparticle patterns

The opportunity to create different patterns of magnetic nanoparticles on surfaces is highly desirable across many technological and biomedical applications. In this paper, this ability is demonstrated for the first time using a computer-controlled aerosol jet printing (AJP) technology. AJP is an emerging digitally driven, non-contact and mask-less printing process which has distinguishing advantages over other patterning technologies as it offers high-resolution and versatile direct-write deposition of a wide range of materials onto a variety of substrates. This research demonstrates the ability of AJP to reliably print large-area, fine-feature patterns of superparamagnetic iron oxide nanoparticles (SPIONs) onto both rigid material (glass) and soft and flexible materials (polydimethylsiloxane (PDMS) films and poly-L-lactic acid (PLLA) nanofilms). Investigation identified and controlled influential process variables which permitted feature sizes in the region of 20 μm to be realised. This method could be employed for a wide range of applications that require a flexible and responsive process that permits high yield and rapid patterning of magnetic material over large areas. As a first proof of concept, we present patterned magnetic nanofilms with enhanced manipulability under external magnetic field gradient control and which are capable of performing complex movements such as rotation and bending, with applicability to soft robotics and biomedical engineering applications.

Towards Autonomous Robotic Minimally Invasive Ultrasound Scanning and Vessel Reconstruction on Non-Planar Surfaces

Autonomous robotic Ultrasound (US) scanning has been the subject of research for more than 2 decades. However, little work has been done to apply this concept into a minimally invasive setting, in which accurate force sensing is generally not available and robot kinematics are unreliable due to the tendon-driven, compliant robot structure. As a result, the adequate orientation of the probe towards the tissue surface remains unknown and the anatomy reconstructed from scan may become highly inaccurate. In this work we present solutions to both of these challenges: an attitude sensor fusion scheme for improved kinematic sensing and a visual, deep learning based algorithm to establish and maintain contact between the organ surface and the US probe. We further introduce a novel scheme to estimate and orient the probe perpendicular to the center line of a vascular structure. Our approach enables, for the first time, to autonomously scan across a non-planar surface and navigate along an anatomical structure with a robotically guided minimally invasive US probe. Our experiments on a vessel phantom with a convex surface confirm a significant improvement of the reconstructed curved vessel geometry, with our approach strongly reducing the mean positional error and variance. In the future, our approach could help identify vascular structures more effectively and help pave the way towards semi-autonomous assistance during partial hepatectomy and the potential to reduce procedure length and complication rates.

Robotic, self-propelled, self-steerable, and disposable colonoscopes: Reality or pipe dream? A state of the art review

Robotic colonoscopes could potentially provide a comfortable, less painful and safer alternative to standard colonoscopy. Recent exciting developments in this field are pushing the boundaries to what is possible in the future. This article provides a comprehensive review of the current work in robotic colonoscopes including self-propelled, steerable and disposable endoscopes that could be alternatives to standard colonoscopy. We discuss the advantages and disadvantages of these systems currently in development and highlight the technical readiness of each system to help the reader understand where and when such systems may be available for routine clinical use and get an idea of where and in which situation they can best be deployed.

Traditional Multiwell Plates and Petri Dishes Limit the Evaluation of the Effects of Ultrasound on Cells In Vitro

Ultrasound accelerates healing in fractured bone; however, the mechanisms responsible are poorly understood. Experimental setups and ultrasound exposures vary or are not adequately characterized across studies, resulting in inter-study variation and difficulty in concluding biological effects. This study investigated experimental variability introduced through the cell culture platform used. Continuous wave ultrasound (45 kHz; 10, 25 or 75 mW/cm², 5 min/d) was applied, using a Duoson device, to Saos-2 cells seeded in multiwell plates or Petri dishes. Pressure field and vibration quantification and finite-element modelling suggested formation of complex interference patterns, resulting in localized displacement and velocity gradients, more pronounced in multiwell plates. Cell experiments revealed lower metabolic activities in both culture platforms at higher ultrasound intensities and absence of mineralization in certain regions of multiwell plates but not in Petri dishes. Thus, the same transducer produced variable results in different cell culture platforms. Analysis on Petri dishes further revealed that higher intensities reduced vinculin expression and distorted cell morphology, while causing mitochondrial and endoplasmic reticulum damage and accumulation of cells in sub-G1 phase, leading to cell death. More defined experimental setups and reproducible ultrasound exposure systems are required to study the real effect of ultrasound on cells for development of effective ultrasound-based therapies not just limited to bone repair and regeneration.

Robotic Autonomy for Magnetic Endoscope Biopsy

Magnetically actuated endoscopes are currently transitioning in to clinical use for procedures such as colonoscopy, presenting numerous benefits over their conventional counterparts. Intelligent and easy-to-use control strategies are an essential part of their clinical effectiveness due to the un-intuitive nature of magnetic field interaction. However, work on developing intelligent control for these devices has mainly been focused on general purpose endoscope navigation. In this work, we investigate the use of autonomous robotic control for magnetic colonoscope intervention via biopsy, another major component of clinical viability. We have developed control strategies with varying levels of robotic autonomy, including semi-autonomous routines for identifying and performing targeted biopsy, as well as random quadrant biopsy. We present and compare the performance of these approaches to magnetic endoscope biopsy against the use of a standard flexible endoscope on bench-top using a colonoscopy training simulator and silicone colon model. The semi-autonomous routines for targeted and random quadrant biopsy were shown to reduce user workload with comparable times to using a standard flexible endoscope.

Active Stabilization of Interventional Tasks Utilizing a Magnetically Manipulated Endoscope

Magnetically actuated robots have become increasingly popular in medical endoscopy over the past decade. Despite the significant improvements in autonomy and control methods, progress within the field of medical magnetic endoscopes has mainly been in the domain of enhanced navigation. Interventional tasks such as biopsy, polyp removal, and clip placement are a major procedural component of endoscopy. Little advancement has been done in this area due to the problem of adequately controlling and stabilizing magnetically actuated endoscopes for interventional tasks. In the present paper we discuss a novel model-based Linear Parameter Varying (LPV) control approach to provide stability during interventional maneuvers. This method linearizes the non-linear dynamic interaction between the external actuation system and the endoscope in a set of equilibria, associated to different distances between the magnetic source and the endoscope, and computes different controllers for each equilibrium. This approach provides the global stability of the overall system and robustness against external disturbances. The performance of the LPV approach is compared to an intelligent teleoperation control method (based on a Proportional Integral Derivative (PID) controller), on the Magnetic Flexible Endoscope (MFE) platform. Four biopsies in different regions of the colon and at two different system equilibria are performed. Both controllers are asked to stabilize the endoscope in the presence of external disturbances (i.e. the introduction of the biopsy forceps through the working channel of the endoscope). The experiments, performed in a benchtop colon simulator, show a maximum reduction of the mean orientation error of the endoscope of 45.8% with the LPV control compared to the PID controller.

Evolutionary Inverse Material Identification: Bespoke Characterization of Soft Materials Using a Metaheuristic Algorithm

The growing interest in soft robotics has resulted in an increased demand for accurate and reliable material modelling. As soft robots experience high deformations, highly nonlinear behavior is possible. Several analytical models that are able to capture this nonlinear behavior have been proposed, however, accurately calibrating them for specific materials and applications can be challenging. Multiple experimental testbeds may be required for material characterization which can be expensive and cumbersome. In this work, we propose an alternative framework for parameter fitting established hyperelastic material models, with the aim of improving their utility in the modelling of soft continuum robots. We define a minimization problem to reduce fitting errors between a soft continuum robot deformed experimentally and its equivalent finite element simulation. The soft material is characterized using four commonly employed hyperelastic material models (Neo Hookean; Mooney–Rivlin; Yeoh; and Ogden). To meet the complexity of the defined problem, we use an evolutionary algorithm to navigate the search space and determine optimal parameters for a selected material model and a specific actuation method, naming this approach as Evolutionary Inverse Material Identification (EIMI). We test the proposed approach with a magnetically actuated soft robot by characterizing two polymers often employed in the field: Dragon Skin™ 10 MEDIUM and Ecoflex™ 00-50. To determine the goodness of the FEM simulation for a specific set of model parameters, we define a function that measures the distance between the mesh of the FEM simulation and the experimental data. Our characterization framework showed an improvement greater than 6% compared to conventional model fitting approaches at different strain ranges based on the benchmark defined. Furthermore, the low variability across the different models obtained using our approach demonstrates reduced dependence on model and strain-range selection, making it well suited to application-specific soft robot modelling.

A decade retrospective of medical robotics research from 2010 to 2020

[Figure: see text].

Establishing key research questions for the implementation of artificial intelligence in colonoscopy: a modified Delphi method

BACKGROUND : Artificial intelligence (AI) research in colonoscopy is progressing rapidly but widespread clinical implementation is not yet a reality. We aimed to identify the top implementation research priorities. METHODS : An established modified Delphi approach for research priority setting was used. Fifteen international experts, including endoscopists and translational computer scientists/engineers, from nine countries participated in an online survey over 9 months. Questions related to AI implementation in colonoscopy were generated as a long-list in the first round, and then scored in two subsequent rounds to identify the top 10 research questions. RESULTS : The top 10 ranked questions were categorized into five themes. Theme 1: clinical trial design/end points (4 questions), related to optimum trial designs for polyp detection and characterization, determining the optimal end points for evaluation of AI, and demonstrating impact on interval cancer rates. Theme 2: technological developments (3 questions), including improving detection of more challenging and advanced lesions, reduction of false-positive rates, and minimizing latency. Theme 3: clinical adoption/integration (1 question), concerning the effective combination of detection and characterization into one workflow. Theme 4: data access/annotation (1 question), concerning more efficient or automated data annotation methods to reduce the burden on human experts. Theme 5: regulatory approval (1 question), related to making regulatory approval processes more efficient. CONCLUSIONS : This is the first reported international research priority setting exercise for AI in colonoscopy. The study findings should be used as a framework to guide future research with key stakeholders to accelerate the clinical implementation of AI in endoscopy.

Feasibility of Fiber Reinforcement Within Magnetically Actuated Soft Continuum Robots

Soft continuum manipulators have the potential to replace traditional surgical catheters; offering greater dexterity with access to previously unfeasible locations for a wide range of interventions including neurological and cardiovascular. Magnetically actuated catheters are of particular interest due to their potential for miniaturization and remote control. Challenges around the operation of these catheters exist however, and one of these occurs when the angle between the actuating field and the local magnetization vector of the catheter exceeds 90°. In this arrangement, deformation generated by the resultant magnetic moment acts to increase magnetic torque, leading to potential instability. This phenomenon can cause unpredictable responses to actuation, particularly for soft, flexible materials. When coupled with the inherent challenges of sensing and localization inside living tissue, this behavior represents a barrier to progress. In this feasibility study we propose and investigate the use of helical fiber reinforcement within magnetically actuated soft continuum manipulators. Using numerical simulation to explore the design space, we optimize fiber parameters to enhance the ratio of torsional to bending stiffness. Through bespoke fabrication of an optimized helix design we validate a single, prototypical two-segment, 40 mm × 6 mm continuum manipulator demonstrating a reduction of 67% in unwanted twisting under actuation.

An Origami-Based Soft Robotic Actuator for Upper Gastrointestinal Endoscopic Applications

Soft pneumatic actuators have been explored for endoscopic applications, but challenges in fabricating complex geometry with desirable dimensions and compliance remain. The addition of an endoscopic camera or tool channel is generally not possible without significant change in the diameter of the actuator. Radial expansion and ballooning of actuator walls during bending is undesirable for endoscopic applications. The inclusion of strain limiting methods like, wound fibre, mesh, or multi-material molding have been explored, but the integration of these design approaches with endoscopic requirements drastically increases fabrication complexity, precluding reliable translation into functional endoscopes. For the first time in soft robotics, we present a multi-channel, single material elastomeric actuator with a fully corrugated design (inspired by origami); offering specific functionality for endoscopic applications. The features introduced in this design include i) fabrication of multi-channel monolithic structure of 8.5 mm diameter, ii) incorporation of the benefits of corrugated design in a single material (i.e., limited radial expansion and improved bending efficiency), iii) design scalability (length and diameter), and iv) incorporation of a central hollow channel for the inclusion of an endoscopic camera. Two variants of the actuator are fabricated which have different corrugated or origami length, i.e., 30 mm and 40 mm respectively). Each of the three actuator channels is evaluated under varying volumetric (0.5 mls-1 and 1.5 mls-1 feed rate) and pressurized control to achieve a similar bending profile with the maximum bending angle of 150°. With the intended use for single use upper gastrointestinal endoscopic application, it is desirable to have linear relationships between actuation and angular position in soft pneumatic actuators with high bending response at low pressures; this is where the origami actuator offers contribution. The soft pneumatic actuator has been demonstrated to achieve a maximum bending angle of 200° when integrated with manually driven endoscope. The simple 3-step fabrication technique produces a complex origami pattern in a soft robotic structure, which promotes low pressure bending through the opening of the corrugation while retaining a small diameter and a central lumen, required for successful endoscope integration.

Ethical implications of AI in robotic surgical training: A Delphi consensus statement

CONTEXT

As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them.

OBJECTIVES

To provide ethical guidance on developing narrow AI applications for surgical training curricula. We define standardised approaches to developing AI driven applications in surgical training that address current recognised ethical implications of utilising AI on surgical data. We aim to describe an ethical approach based on the current evidence, understanding of AI and available technologies, by seeking consensus from an expert committee.

EVIDENCE ACQUISITION

The project was carried out in 3 phases: (1) A steering group was formed to review the literature and summarize current evidence. (2) A larger expert panel convened and discussed the ethical implications of AI application based on the current evidence. A survey was created, with input from panel members. (3) Thirdly, panel-based consensus findings were determined using an online Delphi process to formulate guidance. 30 experts in AI implementation and/or training including clinicians, academics and industry contributed. The Delphi process underwent 3 rounds. Additions to the second and third-round surveys were formulated based on the answers and comments from previous rounds. Consensus opinion was defined as ≥ 80% agreement.

EVIDENCE SYNTHESIS

There was 100% response from all 3 rounds. The resulting formulated guidance showed good internal consistency, with a Cronbach alpha of >0.8. There was 100% consensus that there is currently a lack of guidance on the utilisation of AI in the setting of robotic surgical training. Consensus was reached in multiple areas, including: 1. Data protection and privacy; 2. Reproducibility and transparency; 3. Predictive analytics; 4. Inherent biases; 5. Areas of training most likely to benefit from AI.

CONCLUSIONS

Using the Delphi methodology, we achieved international consensus among experts to develop and reach content validation for guidance on ethical implications of AI in surgical training. Providing an ethical foundation for launching narrow AI applications in surgical training. This guidance will require further validation.

PATIENT SUMMARY

As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them.In this paper we provide guidance on ethical implications of AI in surgical training.

P15: EVALUATING THE INTERNATIONAL USABILITY OF A LOW-COST LAPAROSCOPIC TRAINER IN LOW- AND HIGH-INCOME SETTINGS

Introduction

Limited access to equipment and trained personnel restrict the adoption of laparoscopic surgery globally. There are a wide range of laparoscopic trainers available; however, most of these are not affordable. We propose an ultra-low-cost laparoscopic trainer (Lap-Pack), designed for portability, ease of assembly and compatibility with smart devices. The study aims to evaluate the usability of Lap-Pack as a training tool in low- and high-income settings.

Method

An international usability study was conducted in India and the UK in 2019. The participants (n=60), consisting of senior surgeons (n=18), junior trainees (n=20) and medical students (n=22), were asked to complete two tasks using Lap-Pack. Participants then scored Lap-Pack in a 25-point questionnaire, including a pre-established Face-Validity Criteria and four major evaluation categories – Usability, Camera, View, Material.

Result

Lap-Pack scored highly in Face-Validity with a combined mean score of 4.63 (95%CI: 4.31, 4.95, p <0.05) of a possible 6. In both cohorts, the Usability and Camera categories scored highest, with combined values respectively of 6.10 (95%CI: 6.01, 6.19, p <0.05) and 6.09 (95%CI: 5.88, 6.31, p <0.05) of a possible 7. For both centres, the highest-scoring individual criteria were its light weight and portability.

Conclusion

Overall, Lap-Pack was received positively by medical students and consultants alike, suggesting it is a suitable device for development of skills as part of a larger laparoscopic training curriculum. Its ease of assembly, portability and versatility show promise of increasing access to training opportunities worldwide.

Take-home message

Lap-Pack is an ultra-low-cost, portable laparoscopic simulator featuring compatibility with smart devices designed to help increase access to laparoscopic training worldwide. An international usability study found medical students, junior trainees and senior surgeons rated its usability and camera features highly, suggesting its employability as a laparoscopic training tool on a global scale.

Closed-loop control of soft continuum manipulators under tip follower actuation

Continuum manipulators, inspired by nature, have drawn significant interest within the robotics community. They can facilitate motion within complex environments where traditional rigid robots may be ineffective, while maintaining a reasonable degree of precision. Soft continuum manipulators have emerged as a growing subfield of continuum robotics, with promise for applications requiring high compliance, including certain medical procedures. This has driven demand for new control schemes designed to precisely control these highly flexible manipulators, whose kinematics may be sensitive to external loads, such as gravity. This article presents one such approach, utilizing a rapidly computed kinematic model based on Cosserat rod theory, coupled with sensor feedback to facilitate closed-loop control, for a soft continuum manipulator under tip follower actuation and external loading. This approach is suited to soft manipulators undergoing quasi-static deployment, where actuators apply a follower wrench (i.e., one that is in a constant body frame direction regardless of robot configuration) anywhere along the continuum structure, as can be done in water-jet propulsion. In this article we apply the framework specifically to a tip actuated soft continuum manipulator. The proposed control scheme employs both actuator feedback and pose feedback. The actuator feedback is utilized to both regulate the follower load and to compensate for non-linearities of the actuation system that can introduce kinematic model error. Pose feedback is required to maintain accurate path following. Experimental results demonstrate successful path following with the closed-loop control scheme, with significant performance improvements gained through the use of sensor feedback when compared with the open-loop case.

Guidelines for Robotic Flexible Endoscopy at the Time of COVID-19

Flexible endoscopy involves the insertion of a long narrow flexible tube into the body for diagnostic and therapeutic procedures. In the gastrointestinal (GI) tract, flexible endoscopy plays a major role in cancer screening, surveillance, and treatment programs. As a result of gas insufflation during the procedure, both upper and lower GI endoscopy procedures have been classified as aerosol generating by the guidelines issued by the respective societies during the COVID-19 pandemic-although no quantifiable data on aerosol generation currently exists. Due to the risk of COVID-19 transmission to healthcare workers, most societies halted non-emergency and diagnostic procedures during the lockdown. The long-term implications of stoppage in cancer diagnoses and treatment is predicted to lead to a large increase in preventable deaths. Robotics may play a major role in this field by allowing healthcare operators to control the flexible endoscope from a safe distance and pave a path for protecting healthcare workers through minimizing the risk of virus transmission without reducing diagnostic and therapeutic capacities. This review focuses on the needs and challenges associated with the design of robotic flexible endoscopes for use during a pandemic. The authors propose that a few minor changes to existing platforms or considerations for platforms in development could lead to significant benefits for use during infection control scenarios.

Magnetic flexible endoscope for colonoscopy: an initial learning curve analysis

Background and study aims  Colonoscopy is a technically challenging procedure that requires extensive training to minimize discomfort and avoid trauma due to its drive mechanism. Our academic team developed a magnetic flexible endoscope (MFE) actuated by magnetic coupling under supervisory robotic control to enable a front-pull maneuvering mechanism, with a motion controller user interface, to minimize colon wall stress and potentially reduce the learning curve. We aimed to evaluate this learning curve and understand the user experience. Methods  Five novices (no endoscopy experience), five experienced endoscopists, and five experienced MFE users each performed 40 trials on a model colon using 1:1 block randomization between a pediatric colonoscope (PCF) and the MFE. Cecal intubation (CI) success, time to cecum, and user experience (NASA task load index) were measured. Learning curves were determined by the number of trials needed to reach minimum and average proficiency-defined as the slowest average CI time by an experienced user and the average CI time by all experienced users, respectively. Results  MFE minimum proficiency was achieved by all five novices (median 3.92 trials) and five experienced endoscopists (median 2.65 trials). MFE average proficiency was achieved by four novices (median 14.21 trials) and four experienced endoscopists (median 7.00 trials). PCF minimum and average proficiency levels were achieved by only one novice. Novices' perceived workload with the MFE significantly improved after obtaining minimum proficiency. Conclusions  The MFE has a short learning curve for users with no prior experience-requiring relatively few attempts to reach proficiency and at a reduced perceived workload.

Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation

Early diagnosis of colorectal cancer significantly improves survival. However, over half of cases are diagnosed late due to demand exceeding the capacity for colonoscopy - the "gold standard" for screening. Colonoscopy is limited by the outdated design of conventional endoscopes, associated with high complexity of use, cost and pain. Magnetic endoscopes represent a promising alternative, overcoming drawbacks of pain and cost, but struggle to reach the translational stage as magnetic manipulation is complex and unintuitive. In this work, we use machine vision to develop intelligent and autonomous control of a magnetic endoscope, for the first time enabling non-expert users to effectively perform magnetic colonoscopy in-vivo. We combine the use of robotics, computer vision and advanced control to offer an intuitive and effective endoscopic system. Moreover, we define the characteristics required to achieve autonomy in robotic endoscopy. The paradigm described here can be adopted in a variety of applications where navigation in unstructured environments is required, such as catheters, pancreatic endoscopy, bronchoscopy, and gastroscopy. This work brings alternative endoscopic technologies closer to the translational stage, increasing availability of early-stage cancer treatments.

Parallel Helix Actuators for Soft Robotic Applications

Fabrication of soft pneumatic bending actuators typically involves multiple steps to accommodate the formation of complex internal geometry and the alignment and bonding between soft and inextensible materials. The complexity of these processes intensifies when applied to multi-chamber and small-scale (~10 mm diameter) designs, resulting in poor repeatability. Designs regularly rely on combining multiple prefabricated single chamber actuators or are limited to simple (fixed cross-section) internal chamber geometry, which can result in excessive ballooning and reduced bending efficiency, compelling the addition of constraining materials. In this work, we address existing limitations by presenting a single material molding technique that uses parallel cores with helical features. We demonstrate that through specific orientation and alignment of these internal structures, small diameter actuators may be fabricated with complex internal geometry in a single material-without- additional design-critical steps. The helix design produces wall profiles that restrict radial expansion while allowing compact designs through chamber interlocking, and simplified demolding. We present and evaluate three-chambered designs with varied helical features, demonstrating appreciable bending angles (>180°), three-dimensional workspace coverage, and three-times bodyweight carrying capability. Through application and validation of the constant curvature assumption, forward kinematic models are presented for the actuator and calibrated to account for chamber-specific bending characteristics, resulting in a mean model tip error of 4.1 mm. This simple and inexpensive fabrication technique has potential to be scaled in size and chamber numbers, allowing for application-specific designs for soft, high-mobility actuators especially for surgical, or locomotion applications.

The waterjet necrosectomy device for endoscopic management of pancreatic necrosis: design, development, and preclinical testing (with videos)

BACKGROUND AND AIMS

Endoscopic intervention has emerged as a first-line option for management of symptomatic pancreatic necrosis, yet endoscopic debridement is limited by the lack of dedicated endoscopic tools intended for this purpose. The objectives of this study were to design and build a prototype necrosectomy device compatible for use with a flexible endoscope and capable of selective tissue fragmentation, and to test the prototype in benchtop and porcine models.

METHODS

A novel prototype, named the waterjet necrosectomy device (WAND), was designed and developed, consisting of a single-use disposable endoscopic waterjet instrument capable of waterjet selection and independent tip articulation while fitting through a 2.8-mm working channel of a standard adult upper GI endoscope. Benchtop, ex vivo, and in vivo (porcine) testing was performed in the initial stages of investigation.

RESULTS

The WAND was capable of delivering a continuous waterjet force with a surface pressure of 0.72 bar at a flow rate of 0.37 L/minute. In phase 1 of testing, the WAND was able to achieve complete fragmentation of gelatin as a surrogate for pancreatic necrosis in benchtop testing. In phase 2 of testing, the WAND was able to achieve complete fragmentation of freshly explanted human pancreatic necrosis. In phase 3 of testing for safety in fresh necropsy swine, use of the WAND resulted in no significant tissue trauma, even when irrigation was applied at closer proximity and for more extended duration than would be anticipated in clinical use.

CONCLUSION

The WAND prototype delivers irrigation capable of fragmenting necrotic debris ex vivo and avoiding trauma to healthy nontarget tissue. Planning is underway for first-in-human studies to assess the efficacy and safety of the WAND for endoscopic pancreatic necrosectomy.

Deep learning-based anatomical site classification for upper gastrointestinal endoscopy

Purpose

Upper gastrointestinal (GI) endoscopic image documentation has provided an efficient, low-cost solution to address quality control for endoscopic reporting. The problem is, however, challenging for computer-assisted techniques, because different sites have similar appearances. Additionally, across different patients, site appearance variation may be large and inconsistent. Therefore, according to the British and modified Japanese guidelines, we propose a set of oesophagogastroduodenoscopy (EGD) images to be routinely captured and evaluate its efficiency for deep learning-based classification methods.

Methods

A novel EGD image dataset standardising upper GI endoscopy to several steps is established following landmarks proposed in guidelines and annotated by an expert clinician. To demonstrate the discrimination of proposed landmarks that enable the generation of an automated endoscopic report, we train several deep learning-based classification models utilising the well-annotated images.

Results

We report results for a clinical dataset composed of 211 patients (comprising a total of 3704 EGD images) acquired during routine upper GI endoscopic examinations. We find close agreement between predicted labels using our method and the ground truth labelled by human experts. We observe the limitation of current static image classification scheme for EGD image classification.

Conclusion

Our study presents a framework for developing automated EGD reports using deep learning. We demonstrate that our method is feasible to address EGD image classification and can lead towards improved performance and additionally qualitatively demonstrate its performance on our dataset.

A Compression Valve for Sanitary Control of Fluid-Driven Actuators

With significant research focused on integrating robotics into medical devices, sanitary control of pressurizing fluids in a precise, accurate and customizable way is highly desirable. Current sanitary flow control methods include pinch valves which clamp the pressure line locally to restrict fluid flow; resulting in damage and variable flow characteristics over time. This paper presents a sanitary compression valve based on an eccentric clamping mechanism. The proposed valve distributes clamping forces over a larger area, thereby reducing the plastic deformation and associated influence on flow characteristic. Using the proposed valve, significant reductions in plastic deformation (up to 96%) and flow-rate error (up to 98%) were found, when compared with a standard pinch valve. Additionally, an optimization strategy presents a method for improving linearity and resolution over the working range to suit specific control applications. The valve efficacy has been evaluated through controlled testing of a water jet propelled low-cost endoscopic device. In this case, use of the optimized valve shows a reduction in the average orientation error and its variation, resulting in smoother movement of the endoscopic tip when compared to alternative wet and dry valve solutions. The presented valve offers a customizable solution for sanitary control of fluid driven actuators.

Online Disturbance Estimation for Improving Kinematic Accuracy in Continuum Manipulators

Continuum manipulators are flexible robots which undergo continuous deformation as they are actuated. To describe the elastic deformation of such robots, kinematic models have been developed and successfully applied to a large variety of designs and to various levels of constitutive stiffness. Independent of the design, kinematic models need to be calibrated to best describe the deformation of the manipulator. However, even after calibration, unmodeled effects such as friction, nonlinear elastic and/or spatially varying material properties as well as manufacturing imprecision reduce the accuracy of these models. In this paper, we present a method for improving the accuracy of kinematic models of continuum manipulators through the incorporation of orientation sensor feedback. We achieve this through the use of a "disturbance wrench", which is used to compensate for these unmodeled effects, and is continuously estimated based on orientation sensor feedback as the robot moves through its workspace. The presented method is applied to the HydroJet, a waterjet-actuated soft continuum manipulator, and shows an average of 40% reduction in root mean square position and orientation error in the two most common types of kinematic models for continuum manipulators, a Cosserat rod model and a pseudo-rigid body model.

Adaptive Dynamic Control for Magnetically Actuated Medical Robots

In the present work we discuss a novel dynamic control approach for magnetically actuated robots, by proposing an adaptive control technique, robust towards parametric uncertainties and unknown bounded disturbances. The former generally arise due to partial knowledge of the robots' dynamic parameters, such as inertial factors, the latter are the outcome of unpredictable interaction with unstructured environments. In order to show the application of the proposed approach, we consider controlling the Magnetic Flexible Endoscope (MFE) which is composed of a soft-tethered Internal Permanent Magnet (IPM), actuated with a single External Permanent Magnet (EPM). We provide with experimental analysis to show the possibility of levitating the MFE - one of the most difficult tasks with this platform - in case of partial knowledge of the IPM's dynamics and no knowledge of the tether's behaviour. Experiments in an acrylic tube show a reduction of contact of the 32% compared to non-levitating techniques and 1.75 times faster task completion with respect to previously proposed levitating techniques. More realistic experiments, performed in a colon phantom, show that levitating the capsule achieves faster and smoother exploration and that the minimum time for completing the task is attained by the proposed approach.

Evaluation of a novel low-cost disposable endoscope for visual assessment of the esophagus and stomach in an ex-vivo phantom model

Background and study aims  Our academic lab has developed a novel, low-cost, disposable endoscope for assessment of the esophagus and stomach without need for large equipment or complex electronics. Usability and intuitiveness of the platform are unknown. Methods  The novel endoscope (NE) consists of a high-definition camera, LED module, and three bellows. Compressed air actuates the bellows, producing camera/LED articulation. Insufflation and lens cleaning ports are present. Video can be displayed on any monitor. Total material costs less than $ 35 US. Five novices, five fellows, and five attendings performed five trials using a conventional endoscope and the NE on an upper tract phantom with six gastric landmarks marked. Outcomes included successful identification and time to landmarks; and intuitiveness (NASA task load index; user comments). Results  All landmarks were successfully identified with both endoscopes for all trials (n = 900). Attendings and fellows were quicker with the conventional endoscope when compared to the NE (24.48 v 37.13s; P  < 0.01). There was no significant time difference between platforms for novices ( P  = 0.16). All users found the NE intuitive with low mental and physical demand. Novices reported lower temporal demand and effort when using the NE. Conclusions  The NE was easy to maneuver, intuitive, and successful at visualizing gastric landmarks. All users were pleased with the NE drive mechanism and were successful at visualizing the gastric landmarks in a clinically acceptable time. The novel platform has the potential to facilitate rapid, low-cost, diagnostic assessment of the esophagus and stomach in non-traditional settings - facilitating patient management decisions, minimizing encumbrance, and avoiding cross-contamination.

Miniature Pump for Treatment of Refractory Ascites Based on Local Magnetic Actuation

This paper presents the design, fabrication, and experimental validation of a novel low-cost implantable pump for the treatment of refractory ascites (RA) based on local magnetic actuation (LMA). A reciprocating positive displacement pump displaces liquid unidirectionally through magnetic coupling with a magnetic controller placed on the outside of the patient's body. The proposed solution is intuitive to use given an alignment algorithm that exploits externally placed magnetic field sensors (MFS). The implantable device has a catheter-like shape, is electronic free (no on-board battery), has low fabrication cost (<8 USD), and is able to generate a flow-rate of 3.65 L/h while effectively pumping fluids with various viscosity (1–5.5 cP). RA is commonly treated via costly paracentesis or invasive surgical placement of a transjugular portosystemic shunt (TIPS). The proposed solution can be implanted with minimally invasive techniques and can be used on a daily basis to drain a set amount of liquid, without requiring recurrent hospital visits.

Sensorless Estimation of the Planar Distal Shape of a Tip-Actuated Endoscope

Traditional endoscopes consist of a flexible body and a steerable tip with therapeutic capability. Although prior endoscopes have relied on operator pushing for actuation, recent robotic concepts have relied on the application of a tip force for guidance. In such case, the body of the endoscope can be passive and compliant; however, the body can have significant effect on mechanics of motion and may require modeling. As the endoscope body's shape is often unknown, we have developed an estimation method to recover the approximate distal shape, local to the endoscope's tip, where the tip position and orientation are the only sensed parameters in the system. We leverage a planar dynamic model and extended Kalman filter to obtain a constant-curvature shape estimate of a magnetically guided endoscope. We validated this estimator in both dynamic simulations and on a physical platform. We then used this estimate in a feed-forward control scheme and demonstrated improved trajectory following. This methodology can enable the use of inverse-dynamic control for the tip-based actuation of an endoscope, without the need for shape sensing.

Intelligent magnetic manipulation for gastrointestinal ultrasound

Diagnostic endoscopy in the gastrointestinal tract has remained largely unchanged for decades and is limited to the visualization of the tissue surface, the collection of biopsy samples for diagnoses, and minor interventions such as clipping or tissue removal. In this work, we present the autonomous servoing of a magnetic capsule robot for in-situ, subsurface diagnostics of microanatomy. We investigated and showed the feasibility of closed-loop magnetic control using digitized microultrasound (μUS) feedback; this is crucial for obtaining robust imaging in an unknown and unconstrained environment. We demonstrated the functionality of an autonomous servoing algorithm that uses μUS feedback, both on benchtop trials as well as in-vivo in a porcine model. We have validated this magnetic-μUS servoing in instances of autonomous linear probe motion and were able to locate markers in an agar phantom with 1.0 ± 0.9 mm position accuracy using a fusion of robot localization and μUS image information. This work demonstrates the feasibility of closed-loop robotic μUS imaging in the bowel without the need for either a rigid physical link between the transducer and extracorporeal tools or complex manual manipulation.

Sensitivity Ellipsoids for Force Control of Magnetic Robots with Localization Uncertainty

The navigation of magnetic medical robots typically relies on localizing an actuated, intracorporeal, ferromagnetic body and back-computing a necessary field and gradient that would result in a desired wrench on the device. Uncertainty in this localization degrades the precision of force transmission. Reducing applied force uncertainty may enhance tasks such as in-vivo navigation of miniature robots, actuation of magnetically guided catheters, tissue palpation, as well as simply ensuring a bound on forces applied on sensitive tissue. In this paper, we analyzed the effects of localization noise on force uncertainty by using sensitivity ellipsoids of the magnetic force Jacobian and introduced an algorithm for uncertainty reduction. We validated the algorithm in both a simulation study and in a physical experiment. In simulation, we observed reductions in estimated force uncertainty by factors of up to 2.8 and 3.1 when using one and two actuating magnets, respectively. On a physical platform, we demonstrated a force uncertainty reduction by a factor of up to 2.5 as measured using an external sensor. Being the first consideration of force uncertainty resulting from noisy localization, this work provides a strategy for investigators to minimize uncertainty in magnetic force transmission.

Explicit Model Predictive Control of a Magnetic Flexible Endoscope

In this paper, explicit model predictive control is applied in conjunction with nonlinear optimisation to a magnetically actuated flexible endoscope for the first time. The approach is aimed at computing the motion of the external permanent magnet, given the desired forces and torques. The strategy described here takes advantage of the nonlinear nature of the magnetic actuation and explicitly considers the workspace boundaries, as well as the actuation constraints. Initially, a simplified dynamic model of the tethered capsule, based on the Euler-Lagrange equations is developed. Subsequently, the explicit model predictive control is described and a novel approach for the external magnet positioning, based on a single step, nonlinear optimisation routine, is proposed. Finally, the strategy is implemented on the experimental platform, where bench-top trials are performed on a realistic colon phantom, showing the effectiveness of the technique. The work presented here constitutes an initial exploration for model-based control techniques applied to magnetically manipulated payloads, the techniques described here may be applied to a wide range of devices, including flexible endoscopes and wireless capsules. To our knowledge, this is the first example of advanced closed loop control of magnetic capsules.

Magnetic Levitation for Soft-Tethered Capsule Colonoscopy Actuated With a Single Permanent Magnet: A Dynamic Control Approach

The present letter investigates a novel control approach for magnetically driven soft-tethered capsules for colonoscopy-a potentially painless approach for colon inspection. The focus of this work is on a class of devices composed of a magnetic capsule endoscope actuated by a single external permanent magnet. Actuation is achieved by manipulating the external magnet with a serial manipulator, which in turn produces forces and torques on the internal magnetic capsule. We propose a control strategy which, counteracting gravity, achieves levitation of the capsule. This technique, based on a nonlinear backstepping approach, is able to limit contact with the colon walls, reducing friction, avoiding contact with internal folds, and facilitating the inspection of nonplanar cavities. The approach is validated on an experimental setup, which embodies a general scenario faced in colonoscopy. The experiments show that we can attain 19.5% of contact with the colon wall, compared to the almost 100% of previously proposed approaches. Moreover, we show that the control can be used to navigate the capsule through a more realistic environment-a colon phantom-with reasonable completion time.

Dual-Continuum Design Approach for Intuitive and Low-Cost Upper Gastrointestinal Endoscopy

OBJECTIVE

This paper introduces a methodology to design intuitive, low-cost, and portable devices for visual inspection of the upper gastrointestinal tract.

METHODS

The proposed approach mechanically couples a multi-backbone continuum structure, as the user interface, and a parallel bellows actuator, as the endoscopic tip. Analytical modeling techniques derived from continuum robotics were adopted to describe the endoscopic tip motion from user input, accounting for variations in component size and pneumatic compressibility. The modeling framework was used to improve intuitiveness of user-to-task mapping. This was assessed against a 1:1 target, while ease-of-use was validated using landmark identification tasks performed in a stomach simulator by one expert and ten non-expert users; benchmarked against conventional flexible endoscopy. Pre-clinical validation consisted of comparative trials in in-vivo porcine and human cadaver models.

RESULTS

Target mapping was achieved with an average error of 5° in bending angle. Simulated endoscopies were performed by an expert user successfully, within a time comparable to conventional endoscopy (<1 minute difference). Non-experts using the proposed device achieved visualization of the stomach in a shorter time (9s faster on average) than with a conventional endoscope. The estimated cost is <10 USD and <30 USD for disposable and reusable parts, respectively. Significance and Conclusions: Flexible endoscopes are complex and expensive devices, actuated via non-intuitive cable-driven mechanisms. They frequently break, requiring costly repair, and necessitate a dedicated reprocessing facility to prevent cross contamination. The proposed solution is portable, inexpensive, and easy to use, thus lending itself to disposable use by personnel without formal training in flexible endoscopy.

Enhanced Real-Time Pose Estimation for Closed-Loop Robotic Manipulation of Magnetically Actuated Capsule Endoscopes

Pose estimation methods for robotically guided magnetic actuation of capsule endoscopes have recently enabled trajectory following and automation of repetitive endoscopic maneuvers. However, these methods face significant challenges in their path to clinical adoption including the presence of regions of magnetic field singularity, where the accuracy of the system degrades, and the need for accurate initialization of the capsule's pose. In particular, the singularity problem exists for any pose estimation method that utilizes a single source of magnetic field if the method does not rely on the motion of the magnet to obtain multiple measurements from different vantage points. We analyze the workspace of such pose estimation methods with the use of the point-dipole magnetic field model and show that singular regions exist in areas where the capsule is nominally located during magnetic actuation. Since the dipole model can approximate most magnetic field sources, the problem discussed herein pertains to a wider set of pose estimation techniques. We then propose a novel hybrid approach employing static and time-varying magnetic field sources and show that this system has no regions of singularity. The proposed system was experimentally validated for accuracy, workspace size, update rate and performance in regions of magnetic singularity. The system performed as well or better than prior pose estimation methods without requiring accurate initialization and was robust to magnetic singularity. Experimental demonstration of closed-loop control of a tethered magnetic device utilizing the developed pose estimation technique is provided to ascertain its suitability for robotically guided capsule endoscopy. Hence, advances in closed-loop control and intelligent automation of magnetically actuated capsule endoscopes can be further pursued toward clinical realization by employing this pose estimation system.