BioBERTurk: Exploring Turkish Biomedical Language Model Development Strategies in Low-Resource Setting

Pretrained language models augmented with in-domain corpora show impressive results in biomedicine and clinical Natural Language Processing (NLP) tasks in English. However, there has been minimal work in low-resource languages. Although some pioneering works have shown promising results, many scenarios still need to be explored to engineer effective pretrained language models in biomedicine for low-resource settings. This study introduces the BioBERTurk family and four pretrained models in Turkish for biomedicine. To evaluate the models, we also introduced a labeled dataset to classify radiology reports of head CT examinations. Two parts of the reports, impressions and findings, are evaluated separately to observe the performance of models on longer and less informative text. We compared the models with the Turkish BERT (BERTurk) pretrained with general domain text, multilingual BERT (mBERT), and LSTM+attention-based baseline models. The first model initialized from BERTurk and then further pretrained with biomedical corpus performs statistically better than BERTurk, multilingual BERT, and baseline for both datasets. The second model continues to pretrain the BERTurk model by using only radiology Ph.D. theses to test the effect of task-related text. This model slightly outperformed all models on the impression dataset and showed that using only radiology-related data for continual pre-training could be effective. The third model continues to pretrain by adding radiology theses to the biomedical corpus but does not show a statistically meaningful difference for both datasets. The final model combines radiology and biomedicine corpora with the corpus of BERTurk and pretrains a BERT model from scratch. This model is the worst-performing model of the BioBERT family, even worse than BERTurk and multilingual BERT.

Prediction of malignancy upgrade rate in high-risk breast lesions using an artificial intelligence model: a retrospective study

PURPOSE

High-risk breast lesions (HRLs) are associated with future risk of breast cancer. Considering the pathological subtypes, malignancy upgrade rate differs according to each subtype and depends on various factors such as clinical and radiological features and biopsy method. Using artificial intelligence and machine learning models in breast imaging, evaluations can be made in terms of risk estimation in different research areas. This study aimed to develop a machine learning model to distinguish HRL cases requiring surgical excision from lesions with a low risk of accompanying malignancy.

METHODS

A total of 94 patients who were diagnosed with HRL by image-guided biopsy between January 2008 and March 2020 were included in the study. A structured database was created with clinical and radiological characteristics and histopathological results. A machine learning prediction model was created to make binary classifications of lesions as malignant or benign. Random forest, decision tree, K-nearest neighbors, logistic regression, support vector machine (SVM), and multilayer perceptron machine learning algorithms were used. Among these algorithms, SVM was the most successful. The estimations of malignancy for each case detected by artificial intelligence were combined and statistical analyses were performed.

RESULTS

Considering all cases, the malignancy upgrade rate was 24.5%. A significant association was observed between malignancy upgrade rate and lesion size (P = 0.004), presence of mammography findings (P = 0.022), and breast imaging-reporting and data system category (P = 0.001). A statistically significant association was also found between the artificial intelligence prediction model and malignancy upgrade rate (P < 0.001). With the SVM model, an 84% accuracy and 0.786 area-underthe- curve score were obtained in classifying the data as benign or malignant.

CONCLUSION

Our artificial intelligence model (SVM) can predict HRLs that can be followed up with a lower risk of accompanying malignancy. Unnecessary surgeries can be reduced, or second line vacuum excisions can be performed in HRLs, which are mostly benign, by evaluating on a case-by-case basis, in line with radiology-pathology compatibility and by using an artificial intelligence model.

Mimicking human neuronal pathways in silico: an emergent model on the effective connectivity

We present a novel computational model that detects temporal configurations of a given human neuronal pathway and constructs its artificial replication. This poses a great challenge since direct recordings from individual neurons are impossible in the human central nervous system and therefore the underlying neuronal pathway has to be considered as a black box. For tackling this challenge, we used a branch of complex systems modeling called artificial self-organization in which large sets of software entities interacting locally give rise to bottom-up collective behaviors. The result is an emergent model where each software entity represents an integrate-and-fire neuron. We then applied the model to the reflex responses of single motor units obtained from conscious human subjects. Experimental results show that the model recovers functionality of real human neuronal pathways by comparing it to appropriate surrogate data. What makes the model promising is the fact that, to the best of our knowledge, it is the first realistic model to self-wire an artificial neuronal network by efficiently combining neuroscience with artificial self-organization. Although there is no evidence yet of the model's connectivity mapping onto the human connectivity, we anticipate this model will help neuroscientists to learn much more about human neuronal networks, and could also be used for predicting hypotheses to lead future experiments.