Can Deep Learning Algorithms Help Identify Surgical Workflow and Techniques?

Generating Rare Surgical Events Using CycleGAN: Addressing Lack of Data for Artificial Intelligence Event Recognition

INTRODUCTION

Artificial Intelligence (AI) has shown promise in facilitating surgical video review through automatic recognition of surgical activities/events. There are few public video data sources that demonstrate critical yet rare events which are insufficient to train AI for reliable video event recognition. We suggest that a generative AI algorithm can create artificial massive bleeding images for minimally invasive lobectomy that can be used to augment the current lack of data in this field.

MATERIALS AND METHODS

A generative adversarial network (GAN) algorithm was used (CycleGAN) to generate artificial massive bleeding event images. To train CycleGAN, six videos of minimally invasive lobectomies were utilized from which 1819 frames of nonbleeding instances and 3178 frames of massive bleeding instances were used.

RESULTS

The performance of the CycleGAN algorithm was tested on a new video that was not used during the training process. The trained CycleGAN was able to alter the laparoscopic lobectomy images according to their corresponding massive bleeding images, where the contents of the original images were preserved (e.g., location of tools in the scene) and the style of each image is changed to massive bleeding (i.e., blood automatically added to appropriate locations on the images).

CONCLUSIONS

The result could suggest a promising approach to supplement the lack of data for the rare massive bleeding event that can occur during minimally invasive lobectomy. Future work could be dedicated to developing AI algorithms to identify surgical strategies and actions that potentially lead to massive bleeding and warn surgeons prior to this event occurrence.

Open surgery tool classification and hand utilization using a multi-camera system

PURPOSE

The goal of this work is to use multi-camera video to classify open surgery tools as well as identify which tool is held in each hand. Multi-camera systems help prevent occlusions in open surgery video data. Furthermore, combining multiple views such as a top-view camera covering the full operative field and a close-up camera focusing on hand motion and anatomy may provide a more comprehensive view of the surgical workflow. However, multi-camera data fusion poses a new challenge: A tool may be visible in one camera and not the other. Thus, we defined the global ground truth as the tools being used regardless their visibility. Therefore, tools that are out of the image should be remembered for extensive periods of time while the system responds quickly to changes visible in the video.

METHODS

Participants (n = 48) performed a simulated open bowel repair. A top-view and a close-up cameras were used. YOLOv5 was used for tool and hand detection. A high-frequency LSTM with a 1-second window at 30 frames per second (fps) and a low-frequency LSTM with a 40-second window at 3 fps were used for spatial, temporal, and multi-camera integration.

RESULTS

The accuracy and F1 of the six systems were: top-view (0.88/0.88), close-up (0.81,0.83), both cameras (0.9/0.9), high-fps LSTM (0.92/0.93), low-fps LSTM (0.9/0.91), and our final architecture the multi-camera classifier(0.93/0.94).

CONCLUSION

Since each camera in a multi-camera system may have a partial view of the procedure, we defined a 'global ground truth.' Defining this at the data labeling phase emphasized this requirement at the learning phase, eliminating the need for any heuristic decisions. By combining a system with a high fps and a low fps from the multiple camera array, we improved the classification abilities of the global ground truth.

Fisheye Image Detection of Trees Using Improved YOLOX for Tree Height Estimation

Tree height is an essential indicator in forestry research. This indicator is difficult to measure directly, as well as wind disturbance adds to the measurement difficulty. Therefore, tree height measurement has always been an issue that experts and scholars strive to improve. We propose a tree height measurement method based on tree fisheye images to improve the accuracy of tree height measurements. Our aim is to extract tree height extreme points in fisheye images by proposing an improved lightweight target detection network YOLOX-tiny. We added CBAM attention mechanism, transfer learning, and data enhancement methods to improve the recall rate, F₁ score, AP, and other indicators of YOLOX-tiny. This study improves the detection performance of YOLOX-tiny. The use of deep learning can improve measurement efficiency while ensuring measurement accuracy and stability. The results showed that the highest relative error of tree measurements was 4.06% and the average relative error was 1.62%. The analysis showed that the method performed better at all stages than in previous studies.