Surgomics: personalized prediction of morbidity, mortality and long-term outcome in surgery using machine learning on multimodal data

Parameter-efficient framework for surgical action triplet recognition

PURPOSE

Surgical action triplet recognition is a clinically significant yet challenging task. It provides surgeons with detailed information about surgical scenarios, thereby facilitating clinical decision-making. However, the high similarity among action triplets presents a formidable obstacle to recognition. To enhance accuracy, prior methods necessitated the utilization of larger models, thereby incurring a considerable computational burden.

METHODS

We propose a novel framework known as the Lite and Mega Models (LAM). It comprises a CNN-based fully fine-tuned model (LAM-Lite) and a parameter-efficient fine-tuned model based on the foundation model using Transformer architecture (LAM-Mega). Temporal multi-label data augmentation is introduced for extracting robust class-level features.

RESULTS

Our study demonstrates that LAM outperforms prior methods across various parameter scales on the CholecT50 dataset. Using fewer tunable parameters, LAM achieves a mean average precision (mAP) of 42.1%, a 3.6% improvement over the previous state of the art.

CONCLUSION

Leveraging effective structural design and robust capabilities of the foundational model, our proposed approach successfully strikes a balance between accuracy and computational efficiency. The source code is accessible at https://github.com/Lycus99/LAM .

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

BACKGROUND

Laparoscopic cholecystectomy is a very frequent surgical procedure. However, in an ageing society, less surgical staff will need to perform surgery on patients. Collaborative surgical robots (cobots) could address surgical staff shortages and workload. To achieve context-awareness for surgeon-robot collaboration, the intraoperative action workflow recognition is a key challenge.

METHODS

A surgical process model was developed for intraoperative surgical activities including actor, instrument, action and target in laparoscopic cholecystectomy (excluding camera guidance). These activities, as well as instrument presence and surgical phases were annotated in videos of laparoscopic cholecystectomy performed on human patients (n = 10) and on explanted porcine livers (n = 10). The machine learning algorithm Distilled-Swin was trained on our own annotated dataset and the CholecT45 dataset. The validation of the model was conducted using a fivefold cross-validation approach.

RESULTS

In total, 22,351 activities were annotated with a cumulative duration of 24.9 h of video segments. The machine learning algorithm trained and validated on our own dataset scored a mean average precision (mAP) of 25.7% and a top K = 5 accuracy of 85.3%. With training and validation on our dataset and CholecT45, the algorithm scored a mAP of 37.9%.

CONCLUSIONS

An activity model was developed and applied for the fine-granular annotation of laparoscopic cholecystectomies in two surgical settings. A machine recognition algorithm trained on our own annotated dataset and CholecT45 achieved a higher performance than training only on CholecT45 and can recognize frequently occurring activities well, but not infrequent activities. The analysis of an annotated dataset allowed for the quantification of the potential of collaborative surgical robots to address the workload of surgical staff. If collaborative surgical robots could grasp and hold tissue, up to 83.5% of the assistant's tissue interacting tasks (i.e. excluding camera guidance) could be performed by robots.

[The digital operating room : Chances and risks of artificial intelligence]

At the central workplace of the surgeon the digitalization of the operating room has particular consequences for the surgical work. Starting with intraoperative cross-sectional imaging and sonography, through functional imaging, minimally invasive and robot-assisted surgery up to digital surgical and anesthesiological documentation, the vast majority of operating rooms are now at least partially digitalized. The increasing digitalization of the whole process chain enables not only for the collection but also the analysis of big data. Current research focuses on artificial intelligence for the analysis of intraoperative data as the prerequisite for assistance systems that support surgical decision making or warn of risks; however, these technologies raise new ethical questions for the surgical community that affect the core of surgical work.

Targeted Nanoparticle-Based Diagnostic and Treatment Options for Pancreatic Cancer

Pancreatic ductal adenocarcinoma (PDAC), one of the deadliest cancers, presents significant challenges in diagnosis and treatment due to its aggressive, metastatic nature and lack of early detection methods. A key obstacle in PDAC treatment is the highly complex tumor environment characterized by dense stroma surrounding the tumor, which hinders effective drug delivery. Nanotechnology can offer innovative solutions to these challenges, particularly in creating novel drug delivery systems for existing anticancer drugs for PDAC, such as gemcitabine and paclitaxel. By using customization methods such as incorporating conjugated targeting ligands, tumor-penetrating peptides, and therapeutic nucleic acids, these nanoparticle-based systems enhance drug solubility, extend circulation time, improve tumor targeting, and control drug release, thereby minimizing side effects and toxicity in healthy tissues. Moreover, nanoparticles have also shown potential in precise diagnostic methods for PDAC. This literature review will delve into targeted mechanisms, pathways, and approaches in treating pancreatic cancer. Additional emphasis is placed on the study of nanoparticle-based delivery systems, with a brief mention of those in clinical trials. Overall, the overview illustrates the significant advances in nanomedicine, underscoring its role in transcending the constraints of conventional PDAC therapies and diagnostics.

Prediction of Pathologic Complete Response for Rectal Cancer Based on Pretreatment Factors Using Machine Learning

BACKGROUND

Pathologic complete response after neoadjuvant therapy is an important prognostic indicator for locally advanced rectal cancer and may give insights into which patients might be treated nonoperatively in the future. Existing models for predicting pathologic complete response in the pretreatment setting are limited by small data sets and low accuracy.

OBJECTIVE

We sought to use machine learning to develop a more generalizable predictive model for pathologic complete response for locally advanced rectal cancer.

DESIGN

Patients with locally advanced rectal cancer who underwent neoadjuvant therapy followed by surgical resection were identified in the National Cancer Database from years 2010 to 2019 and were split into training, validation, and test sets. Machine learning techniques included random forest, gradient boosting, and artificial neural network. A logistic regression model was also created. Model performance was assessed using an area under the receiver operating characteristic curve.

SETTINGS

This study used a national, multicenter data set.

PATIENTS

Patients with locally advanced rectal cancer who underwent neoadjuvant therapy and proctectomy.

MAIN OUTCOME MEASURES

Pathologic complete response defined as T0/xN0/x.

RESULTS

The data set included 53,684 patients. Pathologic complete response was experienced by 22.9% of patients. Gradient boosting showed the best performance with an area under the receiver operating characteristic curve of 0.777 (95% CI, 0.773-0.781), compared with 0.684 (95% CI, 0.68-0.688) for logistic regression. The strongest predictors of pathologic complete response were no lymphovascular invasion, no perineural invasion, lower CEA, smaller size of tumor, and microsatellite stability. A concise model including the top 5 variables showed preserved performance.

LIMITATIONS

The models were not externally validated.

CONCLUSIONS

Machine learning techniques can be used to accurately predict pathologic complete response for locally advanced rectal cancer in the pretreatment setting. After fine-tuning a data set including patients treated nonoperatively, these models could help clinicians identify the appropriate candidates for a watch-and-wait strategy. See Video Abstract .

EL CNCER DE RECTO BASADA EN FACTORES PREVIOS AL TRATAMIENTO MEDIANTE EL APRENDIZAJE AUTOMTICO

ANTECEDENTES:La respuesta patológica completa después de la terapia neoadyuvante es un indicador pronóstico importante para el cáncer de recto localmente avanzado y puede dar información sobre qué pacientes podrían ser tratados de forma no quirúrgica en el futuro. Los modelos existentes para predecir la respuesta patológica completa en el entorno previo al tratamiento están limitados por conjuntos de datos pequeños y baja precisión.OBJETIVO:Intentamos utilizar el aprendizaje automático para desarrollar un modelo predictivo más generalizable para la respuesta patológica completa para el cáncer de recto localmente avanzado.DISEÑO:Los pacientes con cáncer de recto localmente avanzado que se sometieron a terapia neoadyuvante seguida de resección quirúrgica se identificaron en la Base de Datos Nacional del Cáncer de los años 2010 a 2019 y se dividieron en conjuntos de capacitación, validación y prueba. Las técnicas de aprendizaje automático incluyeron bosque aleatorio, aumento de gradiente y red neuronal artificial. También se creó un modelo de regresión logística. El rendimiento del modelo se evaluó utilizando el área bajo la curva característica operativa del receptor.ÁMBITO:Este estudio utilizó un conjunto de datos nacional multicéntrico.PACIENTES:Pacientes con cáncer de recto localmente avanzado sometidos a terapia neoadyuvante y proctectomía.PRINCIPALES MEDIDAS DE VALORACIÓN:Respuesta patológica completa definida como T0/xN0/x.RESULTADOS:El conjunto de datos incluyó 53.684 pacientes. El 22,9% de los pacientes experimentaron una respuesta patológica completa. El refuerzo de gradiente mostró el mejor rendimiento con un área bajo la curva característica operativa del receptor de 0,777 (IC del 95%: 0,773 - 0,781), en comparación con 0,684 (IC del 95%: 0,68 - 0,688) para la regresión logística. Los predictores más fuertes de respuesta patológica completa fueron la ausencia de invasión linfovascular, la ausencia de invasión perineural, un CEA más bajo, un tamaño más pequeño del tumor y la estabilidad de los microsatélites. Un modelo conciso que incluye las cinco variables principales mostró un rendimiento preservado.LIMITACIONES:Los modelos no fueron validados externamente.CONCLUSIONES:Las técnicas de aprendizaje automático se pueden utilizar para predecir con precisión la respuesta patológica completa para el cáncer de recto localmente avanzado en el entorno previo al tratamiento. Después de realizar ajustes en un conjunto de datos que incluye pacientes tratados de forma no quirúrgica, estos modelos podrían ayudar a los médicos a identificar a los candidatos adecuados para una estrategia de observar y esperar. (Traducción-Dr. Ingrid Melo ).

Active learning for extracting surgomic features in robot-assisted minimally invasive esophagectomy: a prospective annotation study

BACKGROUND

With Surgomics, we aim for personalized prediction of the patient's surgical outcome using machine-learning (ML) on multimodal intraoperative data to extract surgomic features as surgical process characteristics. As high-quality annotations by medical experts are crucial, but still a bottleneck, we prospectively investigate active learning (AL) to reduce annotation effort and present automatic recognition of surgomic features.

METHODS

To establish a process for development of surgomic features, ten video-based features related to bleeding, as highly relevant intraoperative complication, were chosen. They comprise the amount of blood and smoke in the surgical field, six instruments, and two anatomic structures. Annotation of selected frames from robot-assisted minimally invasive esophagectomies was performed by at least three independent medical experts. To test whether AL reduces annotation effort, we performed a prospective annotation study comparing AL with equidistant sampling (EQS) for frame selection. Multiple Bayesian ResNet18 architectures were trained on a multicentric dataset, consisting of 22 videos from two centers.

RESULTS

In total, 14,004 frames were tag annotated. A mean F1-score of 0.75 ± 0.16 was achieved for all features. The highest F1-score was achieved for the instruments (mean 0.80 ± 0.17). This result is also reflected in the inter-rater-agreement (1-rater-kappa > 0.82). Compared to EQS, AL showed better recognition results for the instruments with a significant difference in the McNemar test comparing correctness of predictions. Moreover, in contrast to EQS, AL selected more frames of the four less common instruments (1512 vs. 607 frames) and achieved higher F1-scores for common instruments while requiring less training frames.

CONCLUSION

We presented ten surgomic features relevant for bleeding events in esophageal surgery automatically extracted from surgical video using ML. AL showed the potential to reduce annotation effort while keeping ML performance high for selected features. The source code and the trained models are published open source.

Prediction model of atrial fibrillation recurrence after Cox-Maze IV procedure in patients with chronic valvular disease and atrial fibrillation based on machine learning algorithm

OBJECTIVES

Atrial fibrillation (AF) is a prevalent cardiac arrhythmia, and Cox-maze IV procedure (CMP-IV) is a commonly employed surgical technique for its treatment. Currently, the risk factors for atrial fibrillation recurrence following CMP-IV remain relatively unclear. In recent years, machine learning algorithms have demonstrated immense potential in enhancing diagnostic accuracy, predicting patient outcomes, and devising personalized treatment strategies. This study aims to evaluate the efficacy of CMP-IV on treating chronic valvular disease with AF, utilize machine learning algorithms to identify potential risk factors for AF recurrence, construct a CMP-IV postoperative AF recurrence prediction model.

METHODS

A total of 555 patients with AF combined with chronic valvular disease, who met the criteria, were enrolled from January 2012 to December 2019 from the Second Xiangya Hospital of Central South University and the Affiliated Xinqiao Hospital of the Army Medical University, with an average age of (57.95±7.96) years, including an AF recurrence group (n=117) and an AF non-recurrence group (n=438). Kaplan-Meier method was used to analyze the sinus rhythm maintenance rate, and 9 machine learning models were developed including random forest, gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost), bootstrap aggregating, logistic regression, categorical boosting (CatBoost), support vector machine, adaptive boosting, and multi-layer perceptron. Five-fold cross-validation and model evaluation indicators [including F1 score, accuracy, precision, recall, and area under the curve (AUC)] were used to evaluate the performance of the models. The 2 best-performing models were selected for further analyze, including feature importance evaluation and Shapley additive explanations (SHAP) analysis, identifying AF recurrence risk factors, and building an AF recurrence risk prediction model.

RESULTS

The 5-year sinus rhythm maintenance rate for the patients was 82.13% (95% CI 78.51% to 85.93%). Among the 9 machine learning models, XGBoost and CatBoost models performed best, with the AUC of 0.768 (95% CI 0.742 to 0.786) and 0.762 (95% CI 0.723 to 0.801), respectively. Feature importance and SHAP analysis showed that duration of AF, preoperative left ventricular ejection fraction, postoperative heart rhythm, preoperative neutrophil-to-lymphocyte ratio, preoperative left atrial diameter, preoperative heart rate, and preoperative white blood cell were important factors for AF recurrence. Conclusion: Machine learning algorithms can be effectively used to identify potential risk factors for AF recurrence after CMP-IV. This study successfuly constructs 2 prediction model which may enhance individualized treatment plans.

Acquisition and usage of robotic surgical data for machine learning analysis

BACKGROUND

The increasing use of robot-assisted surgery (RAS) has led to the need for new methods of assessing whether new surgeons are qualified to perform RAS, without the resource-demanding process of having expert surgeons do the assessment. Computer-based automation and artificial intelligence (AI) are seen as promising alternatives to expert-based surgical assessment. However, no standard protocols or methods for preparing data and implementing AI are available for clinicians. This may be among the reasons for the impediment to the use of AI in the clinical setting.

METHOD

We tested our method on porcine models with both the da Vinci Si and the da Vinci Xi. We sought to capture raw video data from the surgical robots and 3D movement data from the surgeons and prepared the data for the use in AI by a structured guide to acquire and prepare video data using the following steps: 'Capturing image data from the surgical robot', 'Extracting event data', 'Capturing movement data of the surgeon', 'Annotation of image data'.

RESULTS

15 participant (11 novices and 4 experienced) performed 10 different intraabdominal RAS procedures. Using this method we captured 188 videos (94 from the surgical robot, and 94 corresponding movement videos of the surgeons' arms and hands). Event data, movement data, and labels were extracted from the raw material and prepared for use in AI.

CONCLUSION

With our described methods, we could collect, prepare, and annotate images, events, and motion data from surgical robotic systems in preparation for its use in AI.

Machine learning-based analysis of risk factors for atrial fibrillation recurrence after Cox-Maze IV procedure in patients with atrial fibrillation and chronic valvular disease: A retrospective cohort study with a control group

Objectives

To evaluate the efficacy of the Cox-Maze IV procedure (CMP-IV) in combination with valve surgery in patients with both atrial fibrillation (AF) and valvular disease and use machine learning algorithms to identify potential risk factors of AF recurrence.

Methods

A total of 1,026 patients with AF and valvular disease from two hospitals were included in the study. 555 patients received the CMP-IV procedure in addition to valve surgery and left atrial appendage ligation (CMP-IV group), while 471 patients only received valve surgery and left atrial appendage ligation (Non-CMP-IV group). Kaplan-Meier analysis was used to calculate the sinus rhythm maintenance rate. 58 variables were selected as variables for each group and 10 machine learning models were developed respectively. The performance of the models was evaluated using five-fold cross-validation and metrics including F1 score, accuracy, precision, and recall. The four best-performing models for each group were selected for further analysis, including feature importance evaluation and SHAP analysis.

Results

The 5-year sinus rhythm maintenance rate in the CMP-IV group was 82.13% (95% CI: 78.51%, 85.93%), while in the Non-CMP-IV group, it was 13.40% (95% CI: 10.44%, 17.20%). The eXtreme Gradient Boosting (XGBoost), LightGBM, Category Boosting (CatBoost) and Random Fores (RF) models performed the best in the CMP-IV group, with area under the curve (AUC) values of 0.768 (95% CI: 0.742, 0.786), 0.766 (95% CI: 0.744, 0.792), 0.762 (95% CI: 0.723, 0.801), and 0.732 (95% CI: 0.701, 0.763), respectively. In the Non-CMP-IV group, the LightGBM, XGBoost, CatBoost and RF models performed the best, with AUC values of 0.738 (95% CI: 0.699, 0.777), 0.732 (95% CI: 0.694, 0.770), 0.724 (95% CI: 0.668, 0.789), and 0.716 (95% CI: 0.656, 0.774), respectively. Analysis of feature importance and SHAP revealed that duration of AF, preoperative left ventricular ejection fraction, postoperative heart rhythm, preoperative neutrophil-lymphocyte ratio, preoperative left atrial diameter and heart rate were significant factors in AF recurrence.

Conclusion

CMP-IV is effective in treating AF and multiple machine learning models were successfully developed, and several risk factors were identified for AF recurrence, which may aid clinical decision-making and optimize the individual surgical management of AF.