Concept Graph Neural Networks for Surgical Video Understanding

Analysis of relations between objects and comprehension of abstract concepts in the surgical video is important in AI-augmented surgery. However, building models that integrate our knowledge and understanding of surgery remains a challenging endeavor. In this paper, we propose a novel way to integrate conceptual knowledge into temporal analysis tasks using temporal concept graph networks. In the proposed networks, a knowledge graph is incorporated into the temporal video analysis of surgical notions, learning the meaning of concepts and relations as they apply to the data. We demonstrate results in surgical video data for tasks such as verification of the critical view of safety, estimation of the Parkland grading scale as well as recognizing instrument-action-tissue triplets. The results show that our method improves the recognition and detection of complex benchmarks as well as enables other analytic applications of interest.

SAGES consensus recommendations on surgical video data use, structure, and exploration (for research in artificial intelligence, clinical quality improvement, and surgical education)

BACKGROUND

Surgery generates a vast amount of data from each procedure. Particularly video data provides significant value for surgical research, clinical outcome assessment, quality control, and education. The data lifecycle is influenced by various factors, including data structure, acquisition, storage, and sharing; data use and exploration, and finally data governance, which encompasses all ethical and legal regulations associated with the data. There is a universal need among stakeholders in surgical data science to establish standardized frameworks that address all aspects of this lifecycle to ensure data quality and purpose.

METHODS

Working groups were formed, among 48 representatives from academia and industry, including clinicians, computer scientists and industry representatives. These working groups focused on: Data Use, Data Structure, Data Exploration, and Data Governance. After working group and panel discussions, a modified Delphi process was conducted.

RESULTS

The resulting Delphi consensus provides conceptualized and structured recommendations for each domain related to surgical video data. We identified the key stakeholders within the data lifecycle and formulated comprehensive, easily understandable, and widely applicable guidelines for data utilization. Standardization of data structure should encompass format and quality, data sources, documentation, metadata, and account for biases within the data. To foster scientific data exploration, datasets should reflect diversity and remain adaptable to future applications. Data governance must be transparent to all stakeholders, addressing legal and ethical considerations surrounding the data.

CONCLUSION

This consensus presents essential recommendations around the generation of standardized and diverse surgical video databanks, accounting for multiple stakeholders involved in data generation and use throughout its lifecycle. Following the SAGES annotation framework, we lay the foundation for standardization of data use, structure, and exploration. A detailed exploration of requirements for adequate data governance will follow.

SAGES video acquisition framework-analysis of available OR recording technologies by the SAGES AI task force

BACKGROUND

Surgical video recording provides the opportunity to acquire intraoperative data that can subsequently be used for a variety of quality improvement, research, and educational applications. Various recording devices are available for standard operating room camera systems. Some allow for collateral data acquisition including activities of the OR staff, kinematic measurements (motion of surgical instruments), and recording of the endoscopic video streams. Additional analysis through computer vision (CV), which allows software to understand and perform predictive tasks on images, can allow for automatic phase segmentation, instrument tracking, and derivative performance-geared metrics. With this survey, we summarize available surgical video acquisition technologies and associated performance analysis platforms.

METHODS

In an effort promoted by the SAGES Artificial Intelligence Task Force, we surveyed the available video recording technology companies. Of thirteen companies approached, nine were interviewed, each over an hour-long video conference. A standard set of 17 questions was administered. Questions spanned from data acquisition capacity, quality, and synchronization of video with other data, availability of analytic tools, privacy, and access.

RESULTS

Most platforms (89%) store video in full-HD (1080p) resolution at a frame rate of 30 fps. Most (67%) of available platforms store data in a Cloud-based databank as opposed to institutional hard drives. CV powered analysis is featured in some platforms: phase segmentation in 44% platforms, out of body blurring or tool tracking in 33%, and suture time in 11%. Kinematic data are provided by 22% and perfusion imaging in one device.

CONCLUSION

Video acquisition platforms on the market allow for in depth performance analysis through manual and automated review. Most of these devices will be integrated in upcoming robotic surgical platforms. Platform analytic supplementation, including CV, may allow for more refined performance analysis to surgeons and trainees. Most current AI features are related to phase segmentation, instrument tracking, and video blurring.

Artificial intelligence prediction of cholecystectomy operative course from automated identification of gallbladder inflammation

BACKGROUND

Operative courses of laparoscopic cholecystectomies vary widely due to differing pathologies. Efforts to assess intra-operative difficulty include the Parkland grading scale (PGS), which scores inflammation from the initial view of the gallbladder on a 1-5 scale. We investigated the impact of PGS on intra-operative outcomes, including laparoscopic duration, attainment of the critical view of safety (CVS), and gallbladder injury. We additionally trained an artificial intelligence (AI) model to identify PGS.

METHODS

One surgeon labeled surgical phases, PGS, CVS attainment, and gallbladder injury in 200 cholecystectomy videos. We used multilevel Bayesian regression models to analyze the PGS's effect on intra-operative outcomes. We trained AI models to identify PGS from an initial view of the gallbladder and compared model performance to annotations by a second surgeon.

RESULTS

Slightly inflamed gallbladders (PGS-2) minimally increased duration, adding 2.7 [95% compatibility interval (CI) 0.3-7.0] minutes to an operation. This contrasted with maximally inflamed gallbladders (PGS-5), where on average 16.9 (95% CI 4.4-33.9) minutes were added, with 31.3 (95% CI 8.0-67.5) minutes added for the most affected surgeon. Inadvertent gallbladder injury occurred in 25% of cases, with a minimal increase in gallbladder injury observed with added inflammation. However, up to a 28% (95% CI - 2, 63) increase in probability of a gallbladder hole during PGS-5 cases was observed for some surgeons. Inflammation had no substantial effect on whether or not a surgeon attained the CVS. An AI model could reliably (Krippendorff's α = 0.71, 95% CI 0.65-0.77) quantify inflammation when compared to a second surgeon (α = 0.82, 95% CI 0.75-0.87).

CONCLUSIONS

An AI model can identify the degree of gallbladder inflammation, which is predictive of cholecystectomy intra-operative course. This automated assessment could be useful for operating room workflow optimization and for targeted per-surgeon and per-resident feedback to accelerate acquisition of operative skills.

Robotic-assisted minimally invasive esophagectomy (RAMIE) for esophageal cancer training curriculum-a worldwide Delphi consensus study

BACKGROUND

Structured training protocols can safely improve skills prior initiating complex surgical procedures such as robotic-assisted minimally invasive esophagectomy (RAMIE). As no consensus on a training curriculum for RAMIE has been established so far it is our aim to define a protocol for RAMIE with the Delphi consensus methodology.

METHODS

Fourteen worldwide RAMIE experts were defined and were enrolled in this Delphi consensus project. An expert panel was created and three Delphi rounds were performed starting December 2019. Items required for RAMIE included, but were not limited to, virtual reality simulation, wet-lab training, proctoring, and continued monitoring and education. After rating performed by the experts, consensus was defined when a Cronbach alpha of ≥0.80 was reached. If ≥80% of the committee reached a consensus an item was seen as fundamental.

RESULTS

All Delphi rounds were completed by 12-14 (86-100%) participants. After three rounds analyzing our 49-item questionnaire, 40 items reached consensus for a training curriculum of RAMIE.

CONCLUSION

The core principles for RAMIE training were defined. This curriculum may lead to a wider adoption of RAMIE and a reduction in time to reach proficiency.

ADAMTSL2 protein and a soluble biomarker signature identify at-risk non-alcoholic steatohepatitis and fibrosis in adults with NAFLD

BACKGROUND & AIMS

Identifying fibrosis in non-alcoholic fatty liver disease (NAFLD) is essential to predict liver-related outcomes and guide treatment decisions. A protein-based signature of fibrosis could serve as a valuable, non-invasive diagnostic tool. This study sought to identify circulating proteins associated with fibrosis in NAFLD.

METHODS

We used aptamer-based proteomics to measure 4,783 proteins in 2 cohorts (Cohort A and B). Targeted, quantitative assays coupling aptamer-based protein pull down and mass spectrometry (SPMS) validated the profiling results in a bariatric and NAFLD cohort (Cohort C and D, respectively). Generalized linear modeling-logistic regression assessed the ability of candidate proteins to classify fibrosis.

RESULTS

From the multiplex profiling, 16 proteins differed significantly by fibrosis in cohorts A (n = 62) and B (n = 98). Quantitative and robust SPMS assays were developed for 8 proteins and validated in Cohorts C (n = 71) and D (n = 84). The A disintegrin and metalloproteinase with thrombospondin motifs like 2 (ADAMTSL2) protein accurately distinguished non-alcoholic fatty liver (NAFL)/non-alcoholic steatohepatitis (NASH) with fibrosis stage 0-1 (F0-1) from at-risk NASH with fibrosis stage 2-4, with AUROCs of 0.83 and 0.86 in Cohorts C and D, respectively, and from NASH with significant fibrosis (F2-3), with AUROCs of 0.80 and 0.83 in Cohorts C and D, respectively. An 8-protein panel distinguished NAFL/NASH F0-1 from at-risk NASH (AUROCs 0.90 and 0.87 in Cohort C and D, respectively) and NASH F2-3 (AUROCs 0.89 and 0.83 in Cohorts C and D, respectively). The 8-protein panel and ADAMTSL2 protein had superior performance to the NAFLD fibrosis score and fibrosis-4 score.

CONCLUSION

The ADAMTSL2 protein and an 8-protein soluble biomarker panel are highly associated with at-risk NASH and significant fibrosis; they exhibited superior diagnostic performance compared to standard of care fibrosis scores.

LAY SUMMARY

Non-alcoholic fatty liver disease (NAFLD) is one of the most common causes of liver disease worldwide. Diagnosing NAFLD and identifying fibrosis (scarring of the liver) currently requires a liver biopsy. Our study identified novel proteins found in the blood which may identify fibrosis without the need for a liver biopsy.

Distinct Hepatic Gene-Expression Patterns of NAFLD in Patients With Obesity

Approaches to manage nonalcoholic fatty liver disease (NAFLD) are limited by an incomplete understanding of disease pathogenesis. The aim of this study was to identify hepatic gene‐expression patterns associated with different patterns of liver injury in a high‐risk cohort of adults with obesity. Using the NanoString Technologies (Seattle, WA) nCounter assay, we quantified expression of 795 genes, hypothesized to be involved in hepatic fibrosis, inflammation, and steatosis, in liver tissue from 318 adults with obesity. Liver specimens were categorized into four distinct NAFLD phenotypes: normal liver histology (NLH), steatosis only (steatosis), nonalcoholic steatohepatitis without fibrosis (NASH F0), and NASH with fibrosis stage 1‐4 (NASH F1‐F4). One hundred twenty‐five genes were significantly increasing or decreasing as NAFLD pathology progressed. Compared with NLH, NASH F0 was characterized by increased inflammatory gene expression, such as gamma‐interferon‐inducible lysosomal thiol reductase (IFI30) and chemokine (C‐X‐C motif) ligand 9 (CXCL9), while complement and coagulation related genes, such as C9 and complement component 4 binding protein beta (C4BPB), were reduced. In the presence of NASH F1‐F4, extracellular matrix degrading proteinases and profibrotic/scar deposition genes, such as collagens and transforming growth factor beta 1 (TGFB1), were simultaneously increased, suggesting a dynamic state of tissue remodeling. Conclusion: In adults with obesity, distinct states of NAFLD are associated with intrahepatic perturbations in genes related to inflammation, complement and coagulation pathways, and tissue remodeling. These data provide insights into the dynamic pathogenesis of NAFLD in high‐risk individuals.

SAGES consensus recommendations on an annotation framework for surgical video

BACKGROUND

The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration.

METHODS

Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups.

RESULTS

After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established.

CONCLUSIONS

While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

Challenges in surgical video annotation

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Automated operative phase identification in peroral endoscopic myotomy

BACKGROUND

Artificial intelligence (AI) and computer vision (CV) have revolutionized image analysis. In surgery, CV applications have focused on surgical phase identification in laparoscopic videos. We proposed to apply CV techniques to identify phases in an endoscopic procedure, peroral endoscopic myotomy (POEM).

METHODS

POEM videos were collected from Massachusetts General and Showa University Koto Toyosu Hospitals. Videos were labeled by surgeons with the following ground truth phases: (1) Submucosal injection, (2) Mucosotomy, (3) Submucosal tunnel, (4) Myotomy, and (5) Mucosotomy closure. The deep-learning CV model-Convolutional Neural Network (CNN) plus Long Short-Term Memory (LSTM)-was trained on 30 videos to create POEMNet. We then used POEMNet to identify operative phases in the remaining 20 videos. The model's performance was compared to surgeon annotated ground truth.

RESULTS

POEMNet's overall phase identification accuracy was 87.6% (95% CI 87.4-87.9%). When evaluated on a per-phase basis, the model performed well, with mean unweighted and prevalence-weighted F1 scores of 0.766 and 0.875, respectively. The model performed best with longer phases, with 70.6% accuracy for phases that had a duration under 5 min and 88.3% accuracy for longer phases.

DISCUSSION

A deep-learning-based approach to CV, previously successful in laparoscopic video phase identification, translates well to endoscopic procedures. With continued refinements, AI could contribute to intra-operative decision-support systems and post-operative risk prediction.

Desmoid tumor presenting 2 years after elective Roux-en-Y gastric bypass: a case report and review of the literature

Desmoid tumors are rare malignancies derived from myofibroblasts, which can cause significant morbidity due to life-threatening invasion of local structures. Risk factors include familial adenomatous polyposis, antecedent surgical trauma and estrogen exposure. We described a previously healthy 27-year-old female in whom a desmoid tumor developed 2 years after a Roux-en-Y gastric bypass was performed for the treatment of obesity. Computed tomography scan demonstrated a 16-cm complex density intra-abdominal mass. Exploratory laparotomy was performed, revealing a mass firmly adherent to the Roux limb, as well as the jejunojejunostomy and distal portion of the bilopancreatic limb. En bloc resection of the mass and the Roux limb was required, as well as reconstruction of the Roux-en-Y anatomy. This case describes a unique, long-term complication of bariatric surgery, in light of a growing population of patients with altered gastric anatomy.

Risk factors and prevalence of liver disease in review of 2557 routine liver biopsies performed during bariatric surgery

BACKGROUND

Obesity is a known risk factor for nonalcoholic fatty liver disease (NAFLD). However, among individuals undergoing bariatric surgery, the prevalence and risk factors for NAFLD, as well as distinct phenotypes of steatosis, nonalcoholic steatohepatitis (NASH), and fibrosis remain incompletely understood.

OBJECTIVES

To determine the prevalence and risk factors for steatosis, NASH, and fibrosis in individuals undergoing routine bariatric surgery.

SETTING

Academic medical center in the United States.

METHODS

Liver wedge biopsies were performed at the time of surgery between 2001 and 2017. Pathology reports were reviewed, and individuals were grouped by NAFLD phenotype. Covariates including demographic characteristics, co-morbidities, and preoperative laboratory values were compared between groups using Student's t test, Pearson's χ², and logistic regression.

RESULTS

Liver biopsies were obtained in 97.7% of first-time bariatric procedures, representing 2557 patients. Mean age was 45.6 years, mean body mass index was 46.7, and most were non-Hispanic white (76.1%) and female (71.6%). On histologic review 61.2% had steatosis and 30.9% NASH. Fibrosis was identified in 29.3% of individuals, and 7.8% had stage ≥2 fibrosis. On logistic regression, elevated aspartate aminotransferase (odds ratio [OR] 1.87; P < .001) and elevated alanine aminotransferase (OR 1.62; P < .001) were independently associated with fibrosis. Elevated hemoglobin A1C of 5.7% to 6.5% (OR 1.29; P < .01) and >6.5% (OR 3.23; P < .001) were also associated with fibrosis. A similar trend was seen for NASH.

CONCLUSIONS

NASH and/or fibrosis is present in nearly one third of patients undergoing routine bariatric surgery. Risk factors include diabetes, elevated liver enzymes, and diabetes. Risk assessment and aggressive screening should be considered in patients undergoing bariatric surgery.

Artificial Intelligence in Surgery: Promises and Perils

OBJECTIVE

The aim of this review was to summarize major topics in artificial intelligence (AI), including their applications and limitations in surgery. This paper reviews the key capabilities of AI to help surgeons understand and critically evaluate new AI applications and to contribute to new developments.

SUMMARY BACKGROUND DATA

AI is composed of various subfields that each provide potential solutions to clinical problems. Each of the core subfields of AI reviewed in this piece has also been used in other industries such as the autonomous car, social networks, and deep learning computers.

METHODS

A review of AI papers across computer science, statistics, and medical sources was conducted to identify key concepts and techniques within AI that are driving innovation across industries, including surgery. Limitations and challenges of working with AI were also reviewed.

RESULTS

Four main subfields of AI were defined: (1) machine learning, (2) artificial neural networks, (3) natural language processing, and (4) computer vision. Their current and future applications to surgical practice were introduced, including big data analytics and clinical decision support systems. The implications of AI for surgeons and the role of surgeons in advancing the technology to optimize clinical effectiveness were discussed.

CONCLUSIONS

Surgeons are well positioned to help integrate AI into modern practice. Surgeons should partner with data scientists to capture data across phases of care and to provide clinical context, for AI has the potential to revolutionize the way surgery is taught and practiced with the promise of a future optimized for the highest quality patient care.

Posterior reversible encephalopathy syndrome (PRES) after bariatric surgery--a potential consequence associated with rapid withdrawal of antihypertensive medications

A 61-year-old woman with a medical history of intracerebral haemorrhage, hypertension, hyperlipidaemia and carotid stenosis presented to the emergency department with altered mental status 3 weeks after undergoing a vertical sleeve gastrectomy for severe obesity. She presented with a hypertensive emergency and a National Institutes of Health Stroke Scale of 4. CT of the head was unrevealing. MRI showed an abnormal signal within the bilateral posterior border-zone areas, with several foci in the parietal and occipital lobes, and thalami, suggestive of posterior reversible encephalopathy syndrome (PRES). The patient was initially placed on a labetalol drip and her preoperative antihypertensive medications--amlodipine, captopril, triamterene and hydrochlorothiazide--were gradually reintroduced. She returned to her baseline and was stable on discharge. Rapid withdrawal of antihypertensive medications in the early postoperative period of bariatric surgery was the aetiology of PRES in this patient. This case report discusses postoperative care of bariatric surgery patients having hypertension.

Transesophageal endoscopic myotomy (TEEM) for the treatment of achalasia: the United States human experience

BACKGROUND

From our early experience with NOTES, our group has acquired familiarity with transesophageal submucosal dissection and myotomy in swine model, which allowed us to perfect a model to perform purely endoscopic transesophageal myotomy (TEEM) for the treatment of achalasia and apply it into clinical practice. This study was designed to assess the safety, feasibility, and efficacy of TEEM in a series of patients with achalasia.

METHODS

Under institutional review board approval, patients were enrolled on our study, where TEEM was offered as an alternative to laparoscopic or robotic Heller myotomy. The inclusion criteria were patients with achalasia confirmed by esophageal manometry, between age 18 and 50 years, and ASA class 2 or lower. The exclusion criteria were pregnancy, prior esophageal surgery, immunosuppression, coagulopathies, and severe medical comorbidities. The procedures were performed under general anesthesia, with the patient in supine position on positive pressure ventilation. With a GIF-180 (Olympus, Tokyo, Japan) positioned at 10 cm above the GEJ, a mucosotomy was performed at the 2 o'clock position, and a submucosal space was developed caudally creating a controlled submucosal tunnel extending 2 cm distal to the GEJ. Upon completion of this tunnel the gastroesophageal lumen was inspected for mucosal integrity. The scope was then reinserted into the submucosal tunnel and using a triangle-tip knife, myotomy was performed starting at 5 cm above the GEJ and ending at 2 cm below the GEJ. During this process the circular muscle layer of the esophagus was carefully divided with preservation of the longitudinal layer. At the end of the procedure, the mucosal incision was closed longitudinally with endoscopic clips and surgical glue.

RESULTS

Five patients underwent TEEM, with no perioperative complication. All patients reported significant improvement of their dysphagia immediately after the procedure. On the first postoperative day, all barium swallows showed disappearance of the classical bird beak taper, rapid emptying of contrast into the stomach, and absence of leaks. All patients were discharged on the second postoperative day on liquid diet. Two patients reported transient heartburn, which were well controlled with medications. The average preoperative GERD-HRQL was 20, which improved to 11.3 at 7 days postoperative and 2 at 30 days postoperative. To date, three patients have already returned for their 6-month follow-up, reporting adequate swallowing and low LES pressures on esophageal manometry (their mean preoperative LES resting pressure was 36.46 mmHg and residual pressure was 43.16 mmHg, whereas the 6-month follow-up mean LES resting pressure was 10.06 mmHg and residual pressure was 0.43 mmHg).

CONCLUSIONS

TEEM seems to be safe, feasible, and effective for the treatment of patients with achalasia. Long-term data are still necessary for wide-spread utilization of this novel technique.

Reliable gastric closure after natural orifice translumenal endoscopic surgery (NOTES) using a novel automated flexible stapling device

BACKGROUND

Reliable closure of the translumenal incision is one of the main challenges facing natural orifice translumenal endoscopic surgery (NOTES). This study aimed to evaluate the use of an automated flexible stapling device (SurgASSIST) for closure of the gastrotomy incision in a porcine model.

METHODS

A double-channel gastroscope was advanced into the stomach. A gastric wall incision was made, and the endoscope was advanced into the peritoneal cavity. After peritoneoscopy, the endoscope was withdrawn into the stomach. The SurgASSIST stapler was advanced orally into the stomach. The gastrotomy edges were positioned between the opened stapler arms using two endoscopic grasping forceps. Stapler loads with and without a cutting blade were used for gastric closure. After firing of the stapler to close the gastric wall incision, x-ray with contrast was performed to assess for gastric leakage. At the end of the procedure, the animals were killed for a study of closure adequacy.

RESULTS

Four acute animal experiments were performed. The delivery and positioning of the stapler were achieved, with technical difficulties mostly due to a short working length (60 cm) of the device. Firing of the staple delivered four rows of staples. Postmortem examination of pig 1 (when a cutting blade was used) demonstrated full-thickness closure of the gastric wall incision, but the cutting blade caused a transmural hole right at the end of the staple line. For this reason, we stopped using stapler loads with a cutting blade. In the three remaining animals (pigs 2-4), we were able to achieve a full-thickness closure of the gastric wall incision without any complications.

CONCLUSIONS

The flexible stapling device may provide a simple and reliable technique for lumenal closure after NOTES procedures. Further survival studies are currently under way to evaluate the long-term efficacy of gastric closure with the stapler after intraperitoneal interventions.