EndoViT: pretraining vision transformers on a large collection of endoscopic images

PURPOSE

Automated endoscopy video analysis is essential for assisting surgeons during medical procedures, but it faces challenges due to complex surgical scenes and limited annotated data. Large-scale pretraining has shown great success in natural language processing and computer vision communities in recent years. These approaches reduce the need for annotated data, which is of great interest in the medical domain. In this work, we investigate endoscopy domain-specific self-supervised pretraining on large collections of data.

METHODS

To this end, we first collect Endo700k, the largest publicly available corpus of endoscopic images, extracted from nine public Minimally Invasive Surgery (MIS) datasets. Endo700k comprises more than 700,000 images. Next, we introduce EndoViT, an endoscopy-pretrained Vision Transformer (ViT), and evaluate it on a diverse set of surgical downstream tasks.

RESULTS

Our findings indicate that domain-specific pretraining with EndoViT yields notable advantages in complex downstream tasks. In the case of action triplet recognition, our approach outperforms ImageNet pretraining. In semantic segmentation, we surpass the state-of-the-art (SOTA) performance. These results demonstrate the effectiveness of our domain-specific pretraining approach in addressing the challenges of automated endoscopy video analysis.

CONCLUSION

Our study contributes to the field of medical computer vision by showcasing the benefits of domain-specific large-scale self-supervised pretraining for vision transformers. We release both our code and pretrained models to facilitate further research in this direction: https://github.com/DominikBatic/EndoViT .

Deep Learning-Enabled Detection of Pneumoperitoneum in Supine and Erect Abdominal Radiography: Modeling Using Transfer Learning and Semi-Supervised Learning

OBJECTIVE

Detection of pneumoperitoneum using abdominal radiography, particularly in the supine position, is often challenging. This study aimed to develop and externally validate a deep learning model for the detection of pneumoperitoneum using supine and erect abdominal radiography.

MATERIALS AND METHODS

A model that can utilize "pneumoperitoneum" and "non-pneumoperitoneum" classes was developed through knowledge distillation. To train the proposed model with limited training data and weak labels, it was trained using a recently proposed semi-supervised learning method called distillation for self-supervised and self-train learning (DISTL), which leverages the Vision Transformer. The proposed model was first pre-trained with chest radiographs to utilize common knowledge between modalities, fine-tuned, and self-trained on labeled and unlabeled abdominal radiographs. The proposed model was trained using data from supine and erect abdominal radiographs. In total, 191212 chest radiographs (CheXpert data) were used for pre-training, and 5518 labeled and 16671 unlabeled abdominal radiographs were used for fine-tuning and self-supervised learning, respectively. The proposed model was internally validated on 389 abdominal radiographs and externally validated on 475 and 798 abdominal radiographs from the two institutions. We evaluated the performance in diagnosing pneumoperitoneum using the area under the receiver operating characteristic curve (AUC) and compared it with that of radiologists.

RESULTS

In the internal validation, the proposed model had an AUC, sensitivity, and specificity of 0.881, 85.4%, and 73.3% and 0.968, 91.1, and 95.0 for supine and erect positions, respectively. In the external validation at the two institutions, the AUCs were 0.835 and 0.852 for the supine position and 0.909 and 0.944 for the erect position. In the reader study, the readers' performances improved with the assistance of the proposed model.

CONCLUSION

The proposed model trained with the DISTL method can accurately detect pneumoperitoneum on abdominal radiography in both the supine and erect positions.

A Pretrained ELECTRA Model for Kinase-Specific Phosphorylation Site Prediction

Phosphorylation plays a vital role in signal transduction and cell cycle. Identifying and understanding phosphorylation through machine-learning methods has a long history. However, existing methods only learn representations of a protein sequence segment from a labeled dataset itself, which could result in biased or incomplete features, especially for kinase-specific phosphorylation site prediction in which training data are typically sparse. To learn a comprehensive contextual representation of a protein sequence segment for kinase-specific phosphorylation site prediction, we pretrained our model from over 24 million unlabeled sequence fragments using ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately). The pretrained model was applied to kinase-specific site prediction of kinases CDK, PKA, CK2, MAPK, and PKC. The pretrained ELECTRA model achieves 9.02% improvement over BERT and 11.10% improvement over MusiteDeep in the area under the precision-recall curve on the benchmark data.

Effects of pretraining and water temperature on female rats' performance in the Morris water maze

The water maze is a complex spatial task that requires the coordination of multiple systems to perform efficiently. Various factors have been shown to influence performance in this task, including motivational state and prior experience. Although a consistent sex difference has been observed in acquiring the water maze in rats, the contribution of the various factors in female rat performance has not been fully assessed. Therefore, the current study tested the effects of motivation as manipulated by water temperature of the maze and prior experience in the maze on the performance of female rats. It was hypothesized that females pretrained in the maze would perform better than those without exposure to the water maze, regardless of water temperature, but in naïve rats, colder water would improve performance as shown previously in male rats. For pretraining, female rats were taught to find a visible platform in cold (19 °C, 4 trials on one day) and warm (25 °C, 4 trials on one day) water before acquisition trials, with the order of the water temperature randomly assigned. Control rats were not given any training and were naïve to the water maze procedure. Pretrained and control rats were then tested to locate a hidden platform in either cold or warm water for 5 consecutive days. Overall, pretraining had a significant effect on distance, latency, and directness of path to the platform. Water temperature did not show a significant effect on any measure or a significant interaction with pretraining. Thus, while our hypothesis that pretraining would improve performance was supported, the results did not support the prediction that water temperature would also significantly influence performance. These results show that non-spatial pretraining can critically improve the performance of females in acquiring a place strategy for the hidden platform, irrespective of water temperature.