Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

A preliminary exploration into top-down and bottom-up deep-learning approaches to localising neuro-interventional point targets in volumetric MRI

PURPOSE

Point localisation is a critical aspect of many interventional planning procedures, specifically representing anatomical regions of interest or landmarks as individual points. This could be seen as analogous to the problem of visual search in cognitive psychology, in which this search is performed either: bottom-up, constructing increasingly abstract and coarse-resolution features over the entire image; or top-down, using contextual cues from the entire image to refine the scope of the region being investigated. Traditional convolutional neural networks use the former, but it is not clear if this is optimal. This article is a preliminary investigation as to how this motivation affects 3D point localisation in neuro-interventional planning.

METHODS

Two neuro-imaging datasets were collected: one for cortical point localisation for repetitive transcranial magnetic stimulation and the other for sub-cortical anatomy localisation for deep brain stimulation. Four different frameworks were developed using top-down versus bottom-up paradigms as well as representing points as co-ordinates or heatmaps. These networks were applied to point localisation for transcranial magnetic stimulation and subcortical anatomy localisation. These networks were evaluated using cross-validation and a varying number of training datasets to analyse their sensitivity to quantity of training data.

RESULTS

Each network shows increasing performance as the amount of available training data increases, with the co-ordinate-based top-down network consistently outperforming the others. Specifically, the top-down architectures tend to outperform the bottom-up ones. An analysis of their memory consumption also encourages the top-down co-ordinate based architecture as it requires significantly less memory than either bottom-up architectures or those representing their predictions via heatmaps.

CONCLUSION

This paper is a preliminary foray into a fundamental aspect of machine learning architectural design: that of the top-down/bottom-up divide from cognitive psychology. Although there are additional considerations within the particular architectures investigated that could affect these results and the number of architectures investigated is limited, our results do indicate that the less commonly used top-down paradigm could lead to more efficient and effective architectures in the future.

Metrics reloaded: recommendations for image analysis validation

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Virtual reality simulation training improve diagnostic knee arthroscopy and meniscectomy skills: a prospective transfer validity study

PURPOSE

Limited data exist on the actual transfer of skills learned using a virtual reality (VR) simulator for arthroscopy training because studies mainly focused on VR performance improvement and not on transfer to real word (transfer validity). The purpose of this single-blinded, controlled trial was to objectively investigate transfer validity in the context of initial knee arthroscopy training.

METHODS

For this study, 36 junior resident orthopaedic surgeons (postgraduate year one and year two) without prior experience in arthroscopic surgery were enrolled to receive standard knee arthroscopy surgery training (NON-VR group) or standard training plus training on a hybrid virtual reality knee arthroscopy simulator (1 h/month) (VR group). At inclusion, all participants completed a questionnaire on their current arthroscopic technical skills. After 6 months of training, both groups performed three exercises that were evaluated independently by two blinded trainers: i) arthroscopic partial meniscectomy on a bench-top knee simulator; ii) supervised diagnostic knee arthroscopy on a cadaveric knee; and iii) supervised knee partial meniscectomy on a cadaveric knee. Training level was determined with the Arthroscopic Surgical Skill Evaluation Tool (ASSET) score.

RESULTS

Overall, performance (ASSET scores) was better in the VR group than NON-VR group (difference in the global scores: p < 0.001, in bench-top meniscectomy scores: p = 0.03, in diagnostic knee arthroscopy on a cadaveric knee scores: p = 0.04, and in partial meniscectomy on a cadaveric knee scores: p = 0.02). Subgroup analysis by postgraduate year showed that the year-one NON-VR subgroup performed worse than the other subgroups, regardless of the exercise.

CONCLUSION

This study showed the transferability of the technical skills acquired by novice residents on a hybrid virtual reality simulator to the bench-top and cadaveric models. Surgical skill acquired with a VR arthroscopy surgical simulator might safely improve arthroscopy competences in the operating room, also helping to standardise resident training and follow their progress.

LEVEL OF EVIDENCE: 2

SAGES consensus recommendations on surgical video data use, structure, and exploration (for research in artificial intelligence, clinical quality improvement, and surgical education)

BACKGROUND

Surgery generates a vast amount of data from each procedure. Particularly video data provides significant value for surgical research, clinical outcome assessment, quality control, and education. The data lifecycle is influenced by various factors, including data structure, acquisition, storage, and sharing; data use and exploration, and finally data governance, which encompasses all ethical and legal regulations associated with the data. There is a universal need among stakeholders in surgical data science to establish standardized frameworks that address all aspects of this lifecycle to ensure data quality and purpose.

METHODS

Working groups were formed, among 48 representatives from academia and industry, including clinicians, computer scientists and industry representatives. These working groups focused on: Data Use, Data Structure, Data Exploration, and Data Governance. After working group and panel discussions, a modified Delphi process was conducted.

RESULTS

The resulting Delphi consensus provides conceptualized and structured recommendations for each domain related to surgical video data. We identified the key stakeholders within the data lifecycle and formulated comprehensive, easily understandable, and widely applicable guidelines for data utilization. Standardization of data structure should encompass format and quality, data sources, documentation, metadata, and account for biases within the data. To foster scientific data exploration, datasets should reflect diversity and remain adaptable to future applications. Data governance must be transparent to all stakeholders, addressing legal and ethical considerations surrounding the data.

CONCLUSION

This consensus presents essential recommendations around the generation of standardized and diverse surgical video databanks, accounting for multiple stakeholders involved in data generation and use throughout its lifecycle. Following the SAGES annotation framework, we lay the foundation for standardization of data use, structure, and exploration. A detailed exploration of requirements for adequate data governance will follow.

Physical environment of the operating room during cesarean section: A systematic review

INTRODUCTION

Environmental factors in the operating room during cesarean sections are likely important for both women/birthing people and their babies but there is currently a lack of rigorous literature about their evaluation. The principal aim of this study was to systematically examine studies published on the physical environment in the obstetrical operating room during c-sections and its impact on mother and neonate outcomes. The secondary objective was to identify the sensors used to investigate the operating room environment during cesarean sections.

METHODS

In this literature review, we searched MEDLINE a database using the following keywords: Cesarean section AND (operating room environment OR Noise OR Music OR Video recording OR Light level OR Gentle OR Temperature OR Motion Data). Eligible studies had to be published in English or French within the past 10 years and had to investigate the operating room environment during cesarean sections in women. For each study we reported which aspects of the physical environment were investigated in the OR (i.e., noise, music, movement, light or temperature) and the involved sensors.

RESULTS

Of a total of 105 studies screened, we selected 8 articles from title and abstract in PubMed. This small number shows that the field is poorly investigated. The most evaluated environment factors to date are operating room noise and temperature, and the presence of music. Few studies used advanced sensors in the operating room to evaluate environmental factors in a more nuanced and complete way. Two studies concern the sound level, four concern music, one concerns temperature and one analyzed the number of entrances/exits into the OR. No study analyzed light level or more fine-grained movement data.

CONCLUSIONS

Main findings include increase of noise and motion at specific time-points, for example during delivery or anaesthesia; the positive impact of music on parents and staff alike; and that a warmer theatre is better for babies but more uncomfortable for surgeons.

Gaze behavior is related to objective technical skills assessment during virtual reality simulator-based surgical training: a proof of concept

PURPOSE

Simulation-based training allows surgical skills to be learned safely. Most virtual reality-based surgical simulators address technical skills without considering non-technical skills, such as gaze use. In this study, we investigated surgeons' visual behavior during virtual reality-based surgical training where visual guidance is provided. Our hypothesis was that the gaze distribution in the environment is correlated with the simulator's technical skills assessment.

METHODS

We recorded 25 surgical training sessions on an arthroscopic simulator. Trainees were equipped with a head-mounted eye-tracking device. A U-net was trained on two sessions to segment three simulator-specific areas of interest (AoI) and the background, to quantify gaze distribution. We tested whether the percentage of gazes in those areas was correlated with the simulator's scores.

RESULTS

The neural network was able to segment all AoI with a mean Intersection over Union superior to 94% for each area. The gaze percentage in the AoI differed among trainees. Despite several sources of data loss, we found significant correlations between gaze position and the simulator scores. For instance, trainees obtained better procedural scores when their gaze focused on the virtual assistance (Spearman correlation test, N = 7, r = 0.800, p = 0.031).

CONCLUSION

Our findings suggest that visual behavior should be quantified for assessing surgical expertise in simulation-based training environments, especially when visual guidance is provided. Ultimately visual behavior could be used to quantitatively assess surgeons' learning curve and expertise while training on VR simulators, in a way that complements existing metrics.

sEMG-based Motion Recognition for Robotic Surgery Training - A Preliminary Study

Robotic surgery represents a major breakthrough in the evolution of medical technology. Accordingly, efficient skill training and assessment methods should be developed to meet the surgeon's need of acquiring such robotic skills over a relatively short learning curve in a safe manner. Different from conventional training and assessment methods, we aim to explore the surface electromyography (sEMG) signal during the training process in order to obtain semantic and interpretable information to help the trainee better understand and improve his/her training performance. As a preliminary study, motion primitive recognition based on sEMG signal is studied in this work. Using machine learning (ML) technique, it is shown that the sEMG-based motion recognition method is feasible and promising for hand motions along 3 Cartesian axes in the virtual reality (VR) environment of a commercial robotic surgery training platform, which will hence serve as the basis for new robotic surgical skill assessment criterion and training guidance based on muscle activity information. Considering certain motion patterns were less accurately recognized than others, more data collection and deep learning-based analysis will be carried out to further improve the recognition accuracy in future research.

PEg TRAnsfer Workflow recognition challenge report: Do multimodal data improve recognition?

BACKGROUND AND OBJECTIVE

In order to be context-aware, computer-assisted surgical systems require accurate, real-time automatic surgical workflow recognition. In the past several years, surgical video has been the most commonly-used modality for surgical workflow recognition. But with the democratization of robot-assisted surgery, new modalities, such as kinematics, are now accessible. Some previous methods use these new modalities as input for their models, but their added value has rarely been studied. This paper presents the design and results of the "PEg TRAnsfer Workflow recognition" (PETRAW) challenge with the objective of developing surgical workflow recognition methods based on one or more modalities and studying their added value.

METHODS

The PETRAW challenge included a data set of 150 peg transfer sequences performed on a virtual simulator. This data set included videos, kinematic data, semantic segmentation data, and annotations, which described the workflow at three levels of granularity: phase, step, and activity. Five tasks were proposed to the participants: three were related to the recognition at all granularities simultaneously using a single modality, and two addressed the recognition using multiple modalities. The mean application-dependent balanced accuracy (AD-Accuracy) was used as an evaluation metric to take into account class balance and is more clinically relevant than a frame-by-frame score.

RESULTS

Seven teams participated in at least one task with four participating in every task. The best results were obtained by combining video and kinematic data (AD-Accuracy of between 93% and 90% for the four teams that participated in all tasks).

CONCLUSION

The improvement of surgical workflow recognition methods using multiple modalities compared with unimodal methods was significant for all teams. However, the longer execution time required for video/kinematic-based methods(compared to only kinematic-based methods) must be considered. Indeed, one must ask if it is wise to increase computing time by 2000 to 20,000% only to increase accuracy by 3%. The PETRAW data set is publicly available at www.synapse.org/PETRAW to encourage further research in surgical workflow recognition.

Reference-free Bayesian model for pointing errors of typein neurosurgical planning

PURPOSE

Many neurosurgical planning tasks rely on identifying points of interest in volumetric images. Often, these points require significant expertise to identify correctly as, in some cases, they are not visible but instead inferred by the clinician. This leads to a high degree of variability between annotators selecting these points. In particular, errors of type are when the experts fundamentally select different points rather than the same point with some inaccuracy. This complicates research as their mean may not reflect any of the experts' intentions nor the ground truth.

METHODS

We present a regularised Bayesian model for measuring errors of type in pointing tasks. This model is reference-free; in that it does not require a priori knowledge of the ground truth point but instead works on the basis of the level of consensus between multiple annotators. We apply this model to simulated data and clinical data from transcranial magnetic stimulation for chronic pain.

RESULTS

Our model estimates the probabilities of selecting the correct point in the range of 82.6[Formula: see text]88.6% with uncertainties in the range of 2.8[Formula: see text]4.0%. This agrees with the literature where ground truth points are known. The uncertainty has not previously been explored in the literature and gives an indication of the dataset's strength.

CONCLUSIONS

Our reference-free Bayesian framework easily models errors of type in pointing tasks. It allows for clinical studies to be performed with a limited number of annotators where the ground truth is not immediately known, which can be applied widely for better understanding human errors in neurosurgical planning.

A systematic review of annotation for surgical process model analysis in minimally invasive surgery based on video

BACKGROUND

Annotated data are foundational to applications of supervised machine learning. However, there seems to be a lack of common language used in the field of surgical data science. The aim of this study is to review the process of annotation and semantics used in the creation of SPM for minimally invasive surgery videos.

METHODS

For this systematic review, we reviewed articles indexed in the MEDLINE database from January 2000 until March 2022. We selected articles using surgical video annotations to describe a surgical process model in the field of minimally invasive surgery. We excluded studies focusing on instrument detection or recognition of anatomical areas only. The risk of bias was evaluated with the Newcastle Ottawa Quality assessment tool. Data from the studies were visually presented in table using the SPIDER tool.

RESULTS

Of the 2806 articles identified, 34 were selected for review. Twenty-two were in the field of digestive surgery, six in ophthalmologic surgery only, one in neurosurgery, three in gynecologic surgery, and two in mixed fields. Thirty-one studies (88.2%) were dedicated to phase, step, or action recognition and mainly relied on a very simple formalization (29, 85.2%). Clinical information in the datasets was lacking for studies using available public datasets. The process of annotation for surgical process model was lacking and poorly described, and description of the surgical procedures was highly variable between studies.

CONCLUSION

Surgical video annotation lacks a rigorous and reproducible framework. This leads to difficulties in sharing videos between institutions and hospitals because of the different languages used. There is a need to develop and use common ontology to improve libraries of annotated surgical videos.

Behavioural disinhibition in frontotemporal dementia investigated within an ecological framework

Disinhibition is a core symptom in behavioural variant frontotemporal dementia (bvFTD) particularly affecting the daily lives of both patients and caregivers. Yet, characterisation of inhibition disorders is still unclear and management options of these disorders are limited. Questionnaires currently used to investigate behavioural disinhibition do not differentiate between several subtypes of disinhibition, encompass observation biases and lack of ecological validity. In the present work, we explored disinhibition in an original semi-ecological situation, by distinguishing three categories of disinhibition: compulsivity, impulsivity and social disinhibition. First, we measured prevalence and frequency of these disorders in 23 bvFTD patients and 24 healthy controls (HC) in order to identify the phenotypical heterogeneity of disinhibition. Then, we examined the relationships between these metrics, the neuropsychological scores and the behavioural states to propose a more comprehensive view of these neuropsychiatric manifestations. Finally, we studied the context of occurrence of these disorders by investigating environmental factors potentially promoting or reducing them. As expected, we found that patients were more compulsive, impulsive and socially disinhibited than HC. We found that 48% of patients presented compulsivity (e.g., repetitive actions), 48% impulsivity (e.g., oral production) and 100% of the patients group showed social disinhibition (e.g., disregards for rules or investigator). Compulsivity was negatively related with emotions recognition. BvFTD patients were less active if not encouraged in an activity, and their social disinhibition decreased as activity increased. Finally, impulsivity and social disinhibition decreased when patients were asked to focus on a task. Summarising, this study underlines the importance to differentiate subtypes of disinhibition as well as the setting in which they are exhibited, and points to stimulating area for non-pharmacological management.

Virtual reality environments to train soft skills in medical and nursing education: a technical feasibility study between France and Japan

PURPOSE

To meet the urgent and massive training needs of healthcare professionals, the use of digital technologies is proving increasingly relevant, and the rise of digital training platforms shows their usefulness and possibilities. However, despite the impact of these platforms on the medical skills learning, cultural differences are rarely factored in the implementation of these training environments.

METHODS

By using the Scrub Nurse Non-Technical Skills Training System (SunSet), we developed a methodology enabling the adaptation of a virtual reality-based environment and scenarios from French to Japanese cultural and medical practices. We then conducted a technical feasibility study between France and Japan to assess virtual reality simulations acceptance among scrub nurses.

RESULTS

Results in term of acceptance do not reveal major disparity between both populations, and the only emerging significant difference between both groups is on the Behavioral Intention, which is significantly higher for the French scrub nurses. In both cases, participants had a positive outlook.

CONCLUSION

The findings suggest that the methodology we have implemented can be further used in the context of cultural adaptation of non-technical skills learning scenarios in virtual environments for the training and assessment of health care personnel.

A Delphi consensus statement for digital surgery

The use of digital technology is increasing rapidly across surgical specialities, yet there is no consensus for the term ‘digital surgery’. This is critical as digital health technologies present technical, governance, and legal challenges which are unique to the surgeon and surgical patient. We aim to define the term digital surgery and the ethical issues surrounding its clinical application, and to identify barriers and research goals for future practice. 38 international experts, across the fields of surgery, AI, industry, law, ethics and policy, participated in a four-round Delphi exercise. Issues were generated by an expert panel and public panel through a scoping questionnaire around key themes identified from the literature and voted upon in two subsequent questionnaire rounds. Consensus was defined if >70% of the panel deemed the statement important and <30% unimportant. A final online meeting was held to discuss consensus statements. The definition of digital surgery as the use of technology for the enhancement of preoperative planning, surgical performance, therapeutic support, or training, to improve outcomes and reduce harm achieved 100% consensus agreement. We highlight key ethical issues concerning data, privacy, confidentiality and public trust, consent, law, litigation and liability, and commercial partnerships within digital surgery and identify barriers and research goals for future practice. Developers and users of digital surgery must not only have an awareness of the ethical issues surrounding digital applications in healthcare, but also the ethical considerations unique to digital surgery. Future research into these issues must involve all digital surgery stakeholders including patients.

Voxel-based diktiometry: Combining convolutional neural networks with voxel-based analysis and its application in diffusion tensor imaging for Parkinson's disease

Extracting population‐wise information from medical images, specifically in the neurological domain, is crucial to better understanding disease processes and progression. This is frequently done in a whole‐brain voxel‐wise manner, in which a population of patients and healthy controls are registered to a common co‐ordinate space and a statistical test is performed on the distribution of image intensities for each location. Although this method has yielded a number of scientific insights, it is further from clinical applicability as the differences are often small and altogether do not permit for a high‐performing classifier. In this article, we take the opposite approach of using a high‐performing classifier, specifically a traditional convolutional neural network, and then extracting insights from it which can be applied in a population‐wise manner, a method we call voxel‐based diktiometry. We have applied this method to diffusion tensor imaging (DTI) analysis for Parkinson's disease (PD), using the Parkinson's Progression Markers Initiative database. By using the network sensitivity information, we can decompose what elements of the DTI contribute the most to the network's performance, drawing conclusions about diffusion biomarkers for PD that are based on metrics which are not readily expressed in the voxel‐wise approach.

Combining simple interactivity and machine learning: a separable deep learning approach to subthalamic nucleus localization and segmentation in MRI for deep brain stimulation surgical planning

Purpose: Deep brain stimulation (DBS) is an interventional treatment for some neurological and neurodegenerative diseases. For example, in Parkinson's disease, DBS electrodes are positioned at particular locations within the basal ganglia to alleviate the patient's motor symptoms. These interventions depend greatly on a preoperative planning stage in which potential targets and electrode trajectories are identified in a preoperative MRI. Due to the small size and low contrast of targets such as the subthalamic nucleus (STN), their segmentation is a difficult task. Machine learning provides a potential avenue for development, but it has difficulty in segmenting such small structures in volumetric images due to additional problems such as segmentation class imbalance. Approach: We present a two-stage separable learning workflow for STN segmentation consisting of a localization step that detects the STN and crops the image to a small region and a segmentation step that delineates the structure within that region. The goal of this decoupling is to improve accuracy and efficiency and to provide an intermediate representation that can be easily corrected by a clinical user. This correction capability was then studied through a human-computer interaction experiment with seven novice participants and one expert neurosurgeon. Results: Our two-step segmentation significantly outperforms the comparative registration-based method currently used in clinic and approaches the fundamental limit on variability due to the image resolution. In addition, the human-computer interaction experiment shows that the additional interaction mechanism allowed by separating STN segmentation into two steps significantly improves the users' ability to correct errors and further improves performance. Conclusions: Our method shows that separable learning not only is feasible for fully automatic STN segmentation but also leads to improved interactivity that can ease its translation into clinical use.

An ecological approach to identify distinct neural correlates of disinhibition in frontotemporal dementia

Disinhibition is a core symptom of many neurodegenerative diseases, particularly frontotemporal dementia, and is a major cause of stress for caregivers. While a distinction between behavioural and cognitive disinhibition is common, an operational definition of behavioural disinhibition is still missing. Furthermore, conventional assessment of behavioural disinhibition, based on questionnaires completed by the caregivers, often lacks ecological validity. Therefore, their neuroanatomical correlates are non-univocal. In the present work, we used an original behavioural approach in a semi-ecological situation to assess two specific dimensions of behavioural disinhibition: compulsivity and social disinhibition. First, we investigated disinhibition profile in patients compared to controls. Then, to validate our approach, compulsivity and social disinhibition scores were correlated with classic cognitive tests measuring disinhibition (Hayling Test) and social cognition (mini-Social cognition & Emotional Assessment). Finally, we disentangled the anatomical networks underlying these two subtypes of behavioural disinhibition, taking in account the grey (voxel-based morphometry) and white matter (diffusion tensor imaging tractography). We included 17 behavioural variant frontotemporal dementia patients and 18 healthy controls. We identified patients as more compulsive and socially disinhibited than controls. We found that behavioural metrics in the semi-ecological task were related to cognitive performance: compulsivity correlated with the Hayling test and both compulsivity and social disinhibition were associated with the emotion recognition test. Based on voxel-based morphometry and tractography, compulsivity correlated with atrophy in the bilateral orbitofrontal cortex, the right temporal region and subcortical structures, as well as with alterations of the bilateral cingulum and uncinate fasciculus, the right inferior longitudinal fasciculus and the right arcuate fasciculus. Thus, the network of regions related to compulsivity matched the "semantic appraisal" network. Social disinhibition was associated with bilateral frontal atrophy and impairments in the forceps minor, the bilateral cingulum and the left uncinate fasciculus, regions corresponding to the frontal component of the "salience" network. Summarizing, this study validates our semi-ecological approach, through the identification of two subtypes of behavioural disinhibition, and highlights different neural networks underlying compulsivity and social disinhibition. Taken together, these findings are promising for clinical practice by providing a better characterisation of inhibition disorders, promoting their detection and consequently a more adapted management of patients.

Surgical data science - from concepts toward clinical translation

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Validation of virtual reality arthroscopy simulator relevance in characterising experienced surgeons

BACKGROUND

Virtual reality (VR) simulation is particularly suitable for learning arthroscopy skills. Despite significant research, one drawback often outlined is the difficulty in distinguishing performance levels (Construct Validity) in experienced surgeons. Therefore, it seems adequate to search new methods of performance measurements using probe trajectories instead of commonly used metrics.

HYPOTHESIS

It was hypothesized that a larger experience in surgical shoulder arthroscopy would be correlated with better performance on a VR shoulder arthroscopy simulator and that experienced operators would share similar probe trajectories.

MATERIALS & METHODS

After answering to standardized questionnaires, 104 trajectories from 52 surgeons divided into 2 cohorts (26 intermediates and 26 experts) were recorded on a shoulder arthroscopy simulator. The procedure analysed was the "loose body removal" in a right shoulder joint. 10 metrics were computed on the trajectories including procedure duration, overall path length, economy of motion and smoothness. Additionally, Dynamic Time Warping (DTW) was computed on the trajectories for unsupervised hierarchical clustering of the surgeons.

RESULTS

Experts were significantly faster (Median 70.9s Interquartile range [56.4-86.3] vs. 116.1s [82.8-154.2], p<0.01), more fluid (4.6.10⁵mm.s^-3 [3.1.10⁵-7.2.10⁵] vs. 1.5.10⁶mm.s^-3 [2.6.10⁶-3.5.10⁶], p=0.05), and economical in their motion (19.3mm² [9.1-25.9] vs. 33.8mm² [14.8-50.5], p<0.01), but there was no significant difference in performance for path length (671.4mm [503.8-846.1] vs 694.6mm [467.0-1090.1], p=0.62). The DTW clustering differentiates two expertise related groups of trajectories with performance similarities, respectively including 48 expert trajectories for the first group and 52 intermediates and 4 expert trajectories for the second group (Sensitivity of 92%, Specificity of 100%). Hierarchical clustering with DTW significantly identified expert operators from intermediate operators and found trajectory similarities among 24/26 experts.

CONCLUSION

This study demonstrated the Construct Validity of the VR shoulder arthroscopy simulator within groups of experienced surgeons. With new types of metrics simply based on the simulator's raw trajectories, it was possible to significantly distinguish levels of expertise. We demonstrated that clustering analysis with Dynamic Time Warping was able to reliably discriminate between expert operators and intermediate operators.

CLINICAL RELEVANCE

The results have implications for the future of arthroscopic surgical training or post-graduate accreditation programs using virtual reality simulation.

LEVEL OF EVIDENCE

III; prospective comparative study.

Machine learning in deep brain stimulation: A systematic review

Deep Brain Stimulation (DBS) is an increasingly common therapy for a large range of neurological disorders, such as abnormal movement disorders. The effectiveness of DBS in terms of controlling patient symptomatology has made this procedure increasingly used over the past few decades. Concurrently, the popularity of Machine Learning (ML), a subfield of artificial intelligence, has skyrocketed and its influence has more recently extended to medical domains such as neurosurgery. Despite its growing research interest, there has yet to be a literature review specifically on the use of ML in DBS. We have followed a fully systematic methodology to obtain a corpus of 73 papers. In each paper, we identified the clinical application, the type/amount of data used, the method employed, and the validation strategy, further decomposed into 12 different sub-categories. The papers overall illustrated some existing trends in how ML is used in the context of DBS, including the breath of the problem domain and evolving techniques, as well as common frameworks and limitations. This systematic review analyzes at a broad level how ML have been recently used to address clinical problems on DBS, giving insight into how these new computational methods are helping to push the state-of-the-art of functional neurosurgery. DBS clinical workflow is complex, involves many specialists, and raises several clinical issues which have partly been addressed with artificial intelligence. However, several areas remain and those that have been recently addressed with ML are by no means considered "solved" by the community nor are they closed to new and evolving methods.

MIcro-surgical anastomose workflow recognition challenge report

BACKGROUND AND OBJECTIVE

Automatic surgical workflow recognition is an essential step in developing context-aware computer-assisted surgical systems. Video recordings of surgeries are becoming widely accessible, as the operational field view is captured during laparoscopic surgeries. Head and ceiling mounted cameras are also increasingly being used to record videos in open surgeries. This makes videos a common choice in surgical workflow recognition. Additional modalities, such as kinematic data captured during robot-assisted surgeries, could also improve workflow recognition. This paper presents the design and results of the MIcro-Surgical Anastomose Workflow recognition on training sessions (MISAW) challenge whose objective was to develop workflow recognition models based on kinematic data and/or videos.

METHODS

The MISAW challenge provided a data set of 27 sequences of micro-surgical anastomosis on artificial blood vessels. This data set was composed of videos, kinematics, and workflow annotations. The latter described the sequences at three different granularity levels: phase, step, and activity. Four tasks were proposed to the participants: three of them were related to the recognition of surgical workflow at three different granularity levels, while the last one addressed the recognition of all granularity levels in the same model. We used the average application-dependent balanced accuracy (AD-Accuracy) as the evaluation metric. This takes unbalanced classes into account and it is more clinically relevant than a frame-by-frame score.

RESULTS

Six teams participated in at least one task. All models employed deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or a combination of both. The best models achieved accuracy above 95%, 80%, 60%, and 75% respectively for recognition of phases, steps, activities, and multi-granularity. The RNN-based models outperformed the CNN-based ones as well as the dedicated modality models compared to the multi-granularity except for activity recognition.

CONCLUSION

For high levels of granularity, the best models had a recognition rate that may be sufficient for applications such as prediction of remaining surgical time. However, for activities, the recognition rate was still low for applications that can be employed clinically. The MISAW data set is publicly available at http://www.synapse.org/MISAW to encourage further research in surgical workflow recognition.

PassFlow: a multimodal workflow for predicting deep brain stimulation outcomes

PURPOSE

Deep Brain Stimulation (DBS) is a proven therapy for Parkinson's Disease (PD), frequently resulting in an enhancement of motor function. Nonetheless, several undesirable side effects can occur after DBS, which can worsen the quality of life of the patient. Thus, the clinical team has to carefully select patients on whom to perform DBS. Over the past decade, there have been some attempts to relate pre-operative data and DBS clinical outcomes, with most focused on the motor symptomatology. In this paper, we propose a machine learning-based method able to predict a large number of DBS clinical outcomes for PD.

METHODS

We propose a multimodal pipeline, referred to as PassFlow, which predicts 84 clinical post-operative clinical scores. PassFlow is composed of an artificial neural network to compress clinical information, an image processing method from the state-of-the-art to extract morphological biomarkers our of T1 imaging, and an SVM to perform the regressions. We validated PassFlow on 196 PD patients who undergone a DBS.

RESULTS

PassFlow showed correlation coefficients as high as 0.71 and were able to significantly predict 63 out of the 84 scores, outperforming a comparative linear method. The number of metrics that are predicted with this pre-operative information was also found to be correlated with the number of patients with this information available, indicating that the PassFlow method is still actively learning.

CONCLUSION

We presented a novel, machine learning-based pipeline to predict a variety of post-operative clinical outcomes of DBS for PD patients. PassFlow took into account various bio-markers, arising from different data modalities, showing high correlation coefficients for some scores from pre-operative data only. It indicates that many clinical outcomes of DBS can be predicted agnostic to the specific simulation parameters, as PassFlow has been validated without such stimulation-related information.

Automatic cortical target point localisation in MRI for transcranial magnetic stimulation via a multi-resolution convolutional neural network

PURPOSE

Transcranial magnetic stimulation (TMS) is a growing therapy for a variety of psychiatric and neurological disorders that arise from or are modulated by cortical regions of the brain represented by singular 3D target points. These target points are often determined manually with assistance from a pre-operative T1-weighted MRI, although there is growing interest in automatic target point localisation using an atlas. However, both approaches can be time-consuming which has an effect on the clinical workflow, and the latter does not take into account patient variability such as the varying number of cortical gyri where these targets are located.

METHODS

This paper proposes a multi-resolution convolutional neural network for point localisation in MR images for a priori defined points in increasingly finely resolved versions of the input image. This approach is both fast and highly memory efficient, allowing it to run in high-throughput centres, and has the capability of distinguishing between patients with high levels of anatomical variability.

RESULTS

Preliminary experiments have found the accuracy of this network to be [Formula: see text] mm, compared to [Formula: see text] mm for deformable registration and [Formula: see text] mm for a human expert. For most treatment points, the human expert and proposed CNN statistically significantly outperform registration, but neither statistically significantly outperforms the other, suggesting that the proposed network has human-level performance.

CONCLUSIONS

The human-level performance of this network indicates that it can improve TMS planning by automatically localising target points in seconds, avoiding more time-consuming registration or manual point localisation processes. This is particularly beneficial for out-of-hospital centres with limited computational resources where TMS is increasingly being administered.

Ethical implications of AI in robotic surgical training: A Delphi consensus statement

CONTEXT

As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them.

OBJECTIVES

To provide ethical guidance on developing narrow AI applications for surgical training curricula. We define standardised approaches to developing AI driven applications in surgical training that address current recognised ethical implications of utilising AI on surgical data. We aim to describe an ethical approach based on the current evidence, understanding of AI and available technologies, by seeking consensus from an expert committee.

EVIDENCE ACQUISITION

The project was carried out in 3 phases: (1) A steering group was formed to review the literature and summarize current evidence. (2) A larger expert panel convened and discussed the ethical implications of AI application based on the current evidence. A survey was created, with input from panel members. (3) Thirdly, panel-based consensus findings were determined using an online Delphi process to formulate guidance. 30 experts in AI implementation and/or training including clinicians, academics and industry contributed. The Delphi process underwent 3 rounds. Additions to the second and third-round surveys were formulated based on the answers and comments from previous rounds. Consensus opinion was defined as ≥ 80% agreement.

EVIDENCE SYNTHESIS

There was 100% response from all 3 rounds. The resulting formulated guidance showed good internal consistency, with a Cronbach alpha of >0.8. There was 100% consensus that there is currently a lack of guidance on the utilisation of AI in the setting of robotic surgical training. Consensus was reached in multiple areas, including: 1. Data protection and privacy; 2. Reproducibility and transparency; 3. Predictive analytics; 4. Inherent biases; 5. Areas of training most likely to benefit from AI.

CONCLUSIONS

Using the Delphi methodology, we achieved international consensus among experts to develop and reach content validation for guidance on ethical implications of AI in surgical training. Providing an ethical foundation for launching narrow AI applications in surgical training. This guidance will require further validation.

PATIENT SUMMARY

As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them.In this paper we provide guidance on ethical implications of AI in surgical training.

Training situational awareness for scrub nurses: Error recognition in a virtual operating room

Virtual reality simulation provides interesting opportunities to train nurses in a safe environment. While the virtual operating room has proven to be a useful training tool for technical skills, it has been less studied for non-technical skills. This study aimed to assess "Error recognition in a virtual operating room", using a simulation scenario designed to improve situation awareness. Eighteen scrub-nurse students and 8 expert scrub-nurses took part in the experiment. They were immersed in a virtual operating room and reported any errors they observed. There were nineteen errors with various degrees of severity. Measures were retrieved from logs (number of errors, time for detection, movements) and from questionnaires (situation awareness, subjective workload, anxiety and user experience). The results showed that the participants who detected most errors had a higher level of situation awareness, detected high-risk errors faster and felt more immersed in the virtual operating room than those detecting fewer errors. They also felt the workload was lighter and experienced more satisfaction. Students explored the operating room more than experts did and detected more errors, especially those with moderate risk. Debriefings confirmed that virtual simulation is acceptable to trainees and motivates them. It also provides useful and original material for debriefings.

Frontotemporal dementia subtypes based on behavioral inhibition deficits

INTRODUCTION

We aimed to investigate phenotypic heterogeneity in the behavioral variant of frontotemporal dementia (bvFTD) through assessment of inhibition deficits.

METHODS

We assessed occurrences of 16 behavioral inhibition deficits from video recordings of 15 bvFTD patients (early stage) and 15 healthy controls (HC) in an ecological setting. We extracted dimensions of inhibition deficit and analyzed their correlations with cognitive and clinical measures. Using these dimensions, we isolated patient clusters whose atrophy patterns were explored.

RESULTS

After identifying two patterns of inhibition deficit (compulsive automatic behaviors and socially unconventional behaviors), we isolated three behavioral clusters with distinct atrophy patterns. BvFTD-G0 (N = 3), an outlier group, showed severe behavioral disturbances and more severe ventromedial prefrontal cortex/orbitofrontal cortex atrophy. Compared to bvFTD-G1 (N = 6), bvFTD-G2 (N = 6) presented higher anxiety and depression along with less diffuse atrophy especially in midline regions.

DISCUSSION

Identifying clinico-anatomical profiles through behavior observation could help to stratify bvFTD patients for adapted treatments.

Data imputation and compression for Parkinson's disease clinical questionnaires

Medical questionnaires are a valuable source of information but are often difficult to analyse due to both their size and the high possibility of them having missing values. This is a problematic issue in biomedical data science as it may complicate how individual questionnaire data is represented for statistical or machine learning analysis. In this paper, we propose a deeply-learnt residual autoencoder to simultaneously perform non-linear data imputation and dimensionality reduction. We present an extensive analysis of the dynamics of the performance of this autoencoder regarding the compression rate and the proportion of missing values. This method is evaluated on motor and non-motor clinical questionnaires of the Parkinson's Progression Markers Initiative (PPMI) database and consistently outperforms linear coupled imputation and reduction approaches.

Impact of Physician Expertise on Probe Trajectory During Obstetric Ultrasound: A Quantitative Approach for Skill Assessment

INTRODUCTION

The objective of the study was to identify objective metrics to evaluate the significance of a sonographer's expertise on trajectories of ultrasound probe during obstetric ultrasound training procedures.

METHODS

This prospective observational study was conducted at Rennes University Hospital, Department of Obstetrics and Gynecology. We evaluated a panel of sonographers (expert, intermediate, and novice) in performing 3 tasks (brain, heart, and spine) with an obstetric ultrasound simulator (Scantrainer; Medaphor, Cardiff, UK). The trajectories of the probe were logged and recorded by a custom data acquisition software. We computed metrics on the trajectories (duration, path length, average velocity, average acceleration, jerk, working volume) to compare the 3 groups and identify discriminating metrics.

RESULTS

A total of 33 participants were enrolled: 5 experts, 12 intermediates, and 16 novices. Discriminatory metrics were observed among the 3 levels of expertise for duration, velocity, acceleration, and jerk for brain and spine tasks. Working volume was discriminatory for the brain and the heart task. Path length was discriminatory for the brain task.

CONCLUSIONS

Our results suggest a relationship between the sonographer's level of expertise and probe trajectory metrics. Such measurements could be used as an indicator of sonographer proficiency and contribute to automatic analysis of probe trajectory to evaluate the quality of sonography and the sonographer.

Metrics used to evaluate obstetric ultrasound skills on simulators: A systematic review

Obstetric ultrasound simulators are now used for training and evaluating OB/GYN students but there is a lack of literature about evaluation metrics in this setting. In this literature review, we searched MEDLINE and the COCHRANE database using the keywords: (Obstetric OR Fetal) AND (Sonography OR Ultrasound) AND Simulation. Of a total of 263 studies screened, we selected nine articles from the title and the abstract in PubMed, in the past 5 years. Two more article were added from bibliographies. A total of 11 articles were therefore included. from which nine articles were selected from the title and the abstract in PubMed. Two more articles were added from the bibliographies For each study, data about the type of simulation, and the metrics (qualitative or quantitative) used for assessment were collected. The selection of studies shows that evaluation criteria for ultrasound training were qualitative metrics (binary success/fail exercise ; dexterity quoted by an external observer ; Objective Structured Assessment of Ultrasound Skills (OSAUS) Score ; quality of images according to Salomon's score) or quantitative criteria (Accuracy of Biometry - Simulator generated metrics). Most studies used a combination of both. To date, simulator metrics used to discriminate ultrasound skills are performance score quoted by external observers and image quality scoring. Whether probe trajectory metrics can be used to discriminate skills is unknown.

BIAS: Transparent reporting of biomedical image analysis challenges

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

SepaConvNet for Localizing the Subthalamic Nucleus Using One Second Micro-electrode Recordings

Micro-electrode recording (MER) is a powerful way of localizing target structures during neurosurgical procedures such as the implantation of deep brain stimulation electrodes, which is a common treatment for Parkinson's disease and other neurological disorders. While Micro-electrode Recording (MER) provides adjunctive information to guidance assisted by pre-operative imaging, it is not unanimously used in the operating room. The lack of standard use of MER may be in part due to its long duration, which can lead to complications during the operation, or due to high degree of expertise required for their interpretation. Over the past decade, various approaches addressing automating MER analysis for target localization have been proposed, which have mainly focused on feature engineering. While the accuracies obtained are acceptable in certain configurations, one issue with handcrafted MER features is that they do not necessarily capture more subtle differences in MER that could be detected auditorily by an expert neurophysiologist. In this paper, we propose and validate a deep learning-based pipeline for subthalamic nucleus (STN) localization with micro-electrode recordings motivated by the human auditory system. Our proposed Convolutional Neural Network (CNN), referred as SepaConvNet, shows improved accuracy over two comparative networks for locating the STN from one second MER samples.

Striatal shape alteration as a staging biomarker for Parkinson's Disease

Parkinson's Disease provokes alterations of subcortical deep gray matter, leading to subtle changes in the shape of several subcortical structures even before the manifestation of motor and non-motor clinical symptoms. We used an automated registration and segmentation pipeline to measure this structural alteration in one early and one advanced Parkinson's Disease (PD) cohorts, one prodromal stage cohort and one healthy control cohort. These structural alterations are then passed to a machine learning pipeline to classify these populations. Our workflow is able to distinguish different stages of PD based solely on shape analysis of the bilateral caudate nucleus and putamen, with balanced accuracies in the range of 59% to 85%. Furthermore, we compared the significance of each of these subcortical structure, compared the performances of different classifiers on this task, thus quantifying the informativeness of striatal shape alteration as a staging bio-marker for PD.

Impact of the complexity of surgical procedures and intraoperative interruptions on neurosurgical team workload

BACKGROUND

Neurosurgical teams are exposed to various stressors: complexity of surgical procedures, environment, time pressure and interruptions contribute to increasing the perceived workload.

OBJECTIVE

This study aimed to evaluate the impact of interruptions and surgical complexity on neurosurgical team workload.

METHODS

A prospective observational study was conducted on thirty surgical procedures of graduated complexity recorded in our Department of Neurosurgery. A scale was created and used by neurosurgeons to evaluate the perceived complexity of the surgical procedure. Interruptions and severity of interruptions were noted. The workloads of the neurosurgeon, surgical assistant, scrub nurse and circulating nurse were measured on the Surgery Task Load Index (SURG-TLX) at the end of the procedure.

RESULTS

A mean 24.6 interruptions per hour were recorded. The mean interference level of the interruptions was 3.5/7. Mean surgical complexity was 4.3/10. Mean sterile team workload was 43.4/100. The multiple linear regression model showed that sterile team workload increased with surgical complexity (β=6.692, P=.0002) but decreased in spite of increases in the number of interruptions per hour (β=-0.855, P=.027). Neurosurgeon and surgical assistant workload increased with surgical complexity (β=11.53, P<0.0001 and β=7.42, P=0.0007, respectively). Scrub nurse workload decreased in spite of increases in the number of interruptions per hour (β=-1.11, P=.026).

CONCLUSION

Our study suggests positive effects of some interruptions during elective neurosurgical procedures with strong team familiarity.

Predicting the quality of surgical exposure using spatial and procedural features from laparoscopic videos

PURPOSE : Evaluating the quality of surgical procedures is a major concern in minimally invasive surgeries. We propose a bottom-up approach based on the study of Sleeve Gastrectomy procedures, for which we analyze what we assume to be an important indicator of the surgical expertise: the exposure of the surgical scene. We first aim at predicting this indicator with features extracted from the laparoscopic video feed, and second to analyze how the extracted features describing the surgical practice influence this indicator. METHOD : Twenty-nine patients underwent Sleeve Gastrectomy performed by two confirmed surgeons in a monocentric study. Features were extracted from spatial and procedural annotations of the videos, and an expert surgeon evaluated the quality of the surgical exposure at specific instants. The features were used as input of a classifier (linear discriminant analysis followed by a support vector machine) to predict the expertise indicator. Features selected in different configurations of the algorithm were compared to understand their relationships with the surgical exposure and the surgeon's practice. RESULTS : The optimized algorithm giving the best performance used spatial features as input ([Formula: see text]). It also predicted equally the two classes of the indicator, despite their strong imbalance. Analyzing the selection of input features in the algorithm allowed a comparison of different configurations of the algorithm and showed a link between the surgical exposure and the surgeon's practice. CONCLUSION : This preliminary study validates that a prediction of the surgical exposure from spatial features is possible. The analysis of the clusters of feature selected by the algorithm also shows encouraging results and potential clinical interpretations.

Learning procedural skills with a virtual reality simulator: An acceptability study

BACKGROUND

Virtual Reality (VR) simulation has recently been developed and has improved surgical training. Most VR simulators focus on learning technical skills and few on procedural skills. Studies that evaluated VR simulators focused on feasibility, reliability or easiness of use, but few of them used a specific acceptability measurement tool.

OBJECTIVES

The aim of the study was to assess acceptability and usability of a new VR simulator for procedural skill training among scrub nurses, based on the Unified Theory of Acceptance and Use of Technology (UTAUT) model.

PARTICIPANTS

The simulator training system was tested with a convenience sample of 16 non-expert users and 13 expert scrub nurses from the neurosurgery department of a French University Hospital.

METHODS

The scenario was designed to train scrub nurses in the preparation of the instrumentation table for a craniotomy in the operating room (OR).

RESULTS

Acceptability of the VR simulator was demonstrated with no significant difference between expert scrub nurses and non-experts. There was no effect of age, gender or expertise. Workload, immersion and simulator sickness were also rated equally by all participants. Most participants stressed its pedagogical interest, fun and realism, but some of them also regretted its lack of visual comfort.

CONCLUSION

This VR simulator designed to teach surgical procedures can be widely used as a tool in initial or vocational training.

Postural instability and gait disorders after subthalamic nucleus deep brain stimulation in Parkinson's disease: a PET study

INTRODUCTION

Patients with Parkinson's disease sometimes report postural instability and gait disorders (PIGD) after subthalamic nucleus deep brain stimulation (STN-DBS). Whether this is the direct consequence of DBS or the result of natural disease progression is still subject to debate.

OBJECTIVE

To compare changes in brain metabolism during STN-DBS between patients with and without PIGD after surgery.

METHODS

We extracted consecutive patients from a database where all Rennes Hospital patients undergoing STN-DBS are registered, with regular prospective updates of their clinical data. Patients were divided into two groups (PIGD and No PIGD) according to changes after surgery, as measured with a composite score based on the selected Unified Parkinson's Disease Rating Scale items. All patients underwent positron emission tomography with ¹⁸[F]-fluorodeoxyglucose 3 months before and after surgery. We ran an ANOVA with two factors (group: PIGD vs. No PIGD; and phase: preoperative vs. postoperative) on SPM8 to compare changes in brain metabolism between the two groups.

RESULTS

Participants were 56 patients, including 10 in the PIGD group. The two groups had similar baseline (i.e., before surgery) characteristics. We found two clusters of increased metabolism in the PIGD group relative to the No PIGD group: dorsal midbrain/pons, including locomotor mesencephalic region and reticular pontine formation, and right motor cerebellum.

CONCLUSION

We found different metabolic changes during DBS-STN among patients with PIGD, concerning brain regions that are already known to be involved in gait disorders in Parkinson's disease, suggesting that DBS is responsible for the appearance of PIGD.

Automatic annotation of surgical activities using virtual reality environments

PURPOSE

Annotation of surgical activities becomes increasingly important for many recent applications such as surgical workflow analysis, surgical situation awareness, and the design of the operating room of the future, especially to train machine learning methods in order to develop intelligent assistance. Currently, annotation is mostly performed by observers with medical background and is incredibly costly and time-consuming, creating a major bottleneck for the above-mentioned technologies. In this paper, we propose a way to eliminate, or at least limit, the human intervention in the annotation process.

METHODS

Meaningful information about interaction between objects is inherently available in virtual reality environments. We propose a strategy to convert automatically this information into annotations in order to provide as output individual surgical process models.

VALIDATION

We implemented our approach through a peg-transfer task simulator and compared it to manual annotations. To assess the impact of our contribution, we studied both intra- and inter-observer variability.

RESULTS AND CONCLUSION

In average, manual annotations took more than 12 min for 1 min of video to achieve low-level physical activity annotation, whereas automatic annotation is achieved in less than a second for the same video period. We also demonstrated that manual annotation introduced mistakes as well as intra- and inter-observer variability that our method is able to suppress due to the high precision and reproducibility.

Assessment of surgical skills by using surgical navigation in robot-assisted partial nephrectomy

PURPOSE

To assess surgical skills in robot-assisted partial nephrectomy (RAPN) with and without surgical navigation (SN).

METHODS

We employed an SN system that synchronizes the real-time endoscopic image with a virtual reality three-dimensional (3D) model for RAPN and evaluated the skills of two expert surgeons with regard to the identification and dissection of the renal artery (non-SN group, n = 21 [first surgeon n = 9, second surgeon n = 12]; SN group, n = 32 [first surgeon n = 11, second surgeon n = 21]). We converted all movements of the robotic forceps during RAPN into a dedicated vocabulary. Using RAPN videos, we classified all movements of the robotic forceps into direct action (defined as movements of the robotic forceps that directly affect tissues) and connected motion (defined as movements that link actions). In addition, we analyzed the frequency, duration, and occupancy rate of the connected motion.

RESULTS

In the SN group, the R.E.N.A.L nephrometry score was lower (7 vs. 6, P = 0.019) and the time to identify and dissect the renal artery (16 vs. 9 min, P = 0.008) was significantly shorter. The connected motions of inefficient "insert," "pull," and "rotate" motions were significantly improved by SN. SN significantly improved the frequency, duration, and occupancy rate of connected motions of the right hand of the first surgeon and of both hands of the second surgeon. The improvements in connected motions were positively associated with SN for both surgeons.

CONCLUSION

This is the first study to investigate SN for nephron-sparing surgery. SN with 3D models might help improve the connected motions of expert surgeons to ensure efficient RAPN.

Why rankings of biomedical image analysis competitions should be interpreted with care

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Surgical motion analysis using discriminative interpretable patterns

OBJECTIVE

The analysis of surgical motion has received a growing interest with the development of devices allowing their automatic capture. In this context, the use of advanced surgical training systems makes an automated assessment of surgical trainee possible. Automatic and quantitative evaluation of surgical skills is a very important step in improving surgical patient care.

MATERIAL AND METHOD

In this paper, we present an approach for the discovery and ranking of discriminative and interpretable patterns of surgical practice from recordings of surgical motions. A pattern is defined as a series of actions or events in the kinematic data that together are distinctive of a specific gesture or skill level. Our approach is based on the decomposition of continuous kinematic data into a set of overlapping gestures represented by strings (bag of words) for which we compute comparative numerical statistic (tf-idf) enabling the discriminative gesture discovery via its relative occurrence frequency.

RESULTS

We carried out experiments on three surgical motion datasets. The results show that the patterns identified by the proposed method can be used to accurately classify individual gestures, skill levels and surgical interfaces. We also present how the patterns provide a detailed feedback on the trainee skill assessment.

CONCLUSIONS

The proposed approach is an interesting addition to existing learning tools for surgery as it provides a way to obtain a feedback on which parts of an exercise have been used to classify the attempt as correct or incorrect.

Functional atlases for analysis of motor and neuropsychological outcomes after medial globus pallidus and subthalamic stimulation

Anatomical atlases have been developed to improve the targeting of basal ganglia in deep brain stimulation. However, the sole anatomy cannot predict the functional outcome of this surgery. Deep brain stimulation is often a compromise between several functional outcomes: motor, fluency and neuropsychological outcomes in particular. In this study, we have developed anatomo-clinical atlases for the targeting of subthalamic and medial globus pallidus deep brain stimulation. The activated electrode coordinates of 42 patients implanted in the subthalamic nucleus and 29 patients in the medial globus pallidus were studied. The atlas was built using the representation of the volume of tissue theoretically activated by the stimulation. The UPDRS score was used to represent the motor outcome. The Stroop test was represented as well as semantic and phonemic fluencies. For the subthalamic nucleus, best motor outcomes were obtained when the supero-lateral part of the nucleus was stimulated whereas the semantic fluency was impaired in this same region. For the medial globus pallidus, best outcomes were obtained when the postero ventral part of the nucleus was stimulated whereas the phonemic fluency was impaired in this same region. There was no significant neuropsychological impairment. We have proposed new anatomo-clinical atlases to visualize the motor and neuropsychological consequences at 6 months of subthalamic nucleus and pallidal stimulation in patients with Parkinson's disease.