Multi-task deep learning for medical image computing and analysis: A review

A multi-task self-supervised approach for mass detection in automated breast ultrasound using double attention recurrent residual U-Net

Breast cancer is the most common and lethal cancer among women worldwide. Early detection using medical imaging technologies can significantly improve treatment outcomes. Automated breast ultrasound, known as ABUS, offers more advantages compared to traditional mammography and has recently gained considerable attention. However, reviewing hundreds of ABUS slices imposes a high workload on radiologists, increasing review time and potentially leading to diagnostic errors. Consequently, there is a strong need for efficient computer-aided detection, CADe, systems. In recent years, researchers have proposed deep learning-based CADe systems to enhance mass detection accuracy. However, these methods are highly dependent on the number of training samples and often struggle to balance detection accuracy with the false positive rate. To reduce the workload for radiologists and achieve high detection sensitivities with low false positive rates, this study introduces a novel CADe system based on a self-supervised framework that leverages unannotated ABUS datasets to improve detection results. The proposed framework is integrated into an innovative 3-D convolutional neural network called DATTR2U-Net, which employs a multi-task learning approach to simultaneously train inpainting and denoising pretext tasks. A fully convolutional network is then attached to the DATTR2U-Net for the detection task. The proposed method is validated on the TDSCABUS public dataset, demonstrating promising detection results with a recall of 0.7963 and a false positive rate of 5.67 per volume that signifies its potential to improve detection accuracy while reducing workload for radiologists. The code is available at: github.com/Pooryamn/SSL_ABUS.

Multitask learning in minimally invasive surgical vision: A review

Minimally invasive surgery (MIS) has revolutionized many procedures and led to reduced recovery time and risk of patient injury. However, MIS poses additional complexity and burden on surgical teams. Data-driven surgical vision algorithms are thought to be key building blocks in the development of future MIS systems with improved autonomy. Recent advancements in machine learning and computer vision have led to successful applications in analysing videos obtained from MIS with the promise of alleviating challenges in MIS videos. Surgical scene and action understanding encompasses multiple related tasks that, when solved individually, can be memory-intensive, inefficient, and fail to capture task relationships. Multitask learning (MTL), a learning paradigm that leverages information from multiple related tasks to improve performance and aid generalization, is well-suited for fine-grained and high-level understanding of MIS data. This review provides a narrative overview of the current state-of-the-art MTL systems that leverage videos obtained from MIS. Beyond listing published approaches, we discuss the benefits and limitations of these MTL systems. Moreover, this manuscript presents an analysis of the literature for various application fields of MTL in MIS, including those with large models, highlighting notable trends, new directions of research, and developments.

An ensemble deep learning framework for multi-class LncRNA subcellular localization with innovative encoding strategy

BACKGROUND

Long non-coding RNA (LncRNA) play pivotal roles in various cellular processes, and elucidating their subcellular localization can offer crucial insights into their functional significance. Accurate prediction of lncRNA subcellular localization is of paramount importance. Despite numerous computational methods developed for this purpose, existing approaches still encounter challenges stemming from the complexity of data representation and the difficulty in capturing nucleotide distribution information within sequences.

RESULTS

In this study, we propose a novel deep learning-based model, termed MGBLncLoc, which incorporates a unique multi-class encoding technique known as generalized encoding based on the Distribution Density of Multi-Class Nucleotide Groups (MCD-ND). This encoding approach enables more precise reflection of nucleotide distributions, distinguishing between constant and discriminative regions within sequences, thereby enhancing prediction performance. Additionally, our deep learning model integrates advanced neural network modules, including Multi-Dconv Head Transposed Attention, Gated-Dconv Feed-forward Network, Convolutional Neural Network, and Bidirectional Gated Recurrent Unit, to comprehensively exploit sequence features of lncRNA.

CONCLUSIONS

Comparative analysis against commonly used sequence feature encoding methods and existing prediction models validates the effectiveness of MGBLncLoc, demonstrating superior performance. This research offers novel insights and effective solutions for predicting lncRNA subcellular localization, thereby providing valuable support for related biological investigations.

NMTNet: A Multi-task Deep Learning Network for Joint Segmentation and Classification of Breast Tumors

Segmentation and classification of breast tumors are two critical tasks since they provide significant information for computer-aided breast cancer diagnosis. Combining these tasks leverages their intrinsic relevance to enhance performance, but the variability and complexity of tumor characteristics remain challenging. We propose a novel multi-task deep learning network (NMTNet) for the joint segmentation and classification of breast tumors, which is based on a convolutional neural network (CNN) and U-shaped architecture. It mainly comprises a shared encoder, a multi-scale fusion channel refinement (MFCR) module, a segmentation branch, and a classification branch. First, ResNet18 is used as the backbone network in the encoding part to enhance the feature representation capability. Then, the MFCR module is introduced to enrich the feature depth and diversity. Besides, the segmentation branch combines a lesion region enhancement (LRE) module between the encoder and decoder parts, aiming to capture more detailed texture and edge information of irregular tumors to improve segmentation accuracy. The classification branch incorporates a fine-grained classifier that reuses valuable segmentation information to discriminate between benign and malignant tumors. The proposed NMTNet is evaluated on both ultrasound and magnetic resonance imaging datasets. It achieves segmentation dice scores of 90.30% and 91.50%, and Jaccard indices of 84.70% and 88.10% for each dataset, respectively. And the classification accuracy scores are 87.50% and 99.64% for the corresponding datasets, respectively. Experimental results demonstrate the superiority of NMTNet over state-of-the-art methods on breast tumor segmentation and classification tasks.

Development of a diagnostic classification model for lateral cephalograms based on multitask learning

OBJECTIVES

This study aimed to develop a cephalometric classification method based on multitask learning for eight diagnostic classifications.

METHODS

This study was retrospective. A total of 3,310 lateral cephalograms were collected to construct a dataset. Eight clinical classifications were employed, including sagittal and vertical skeletal facial patterns, maxillary and mandibular anteroposterior positions, inclinations of upper and lower incisors, as well as their anteroposterior positions. The images were manually annotated for initially classification, which was verified by senior orthodontists. The data were randomly divided into training, validation, and test sets at a ratio of approximately 8:1:1. The multitask learning classification model was constructed based on the ResNeXt50_32 × 4d network and consisted of shared layers and task-specific layers. The performance of the model was evaluated using classification accuracy, precision, sensitivity, specificity and area under the curve (AUC).

RESULTS

This model could perform eight clinical diagnostic classifications on cephalograms within an average of 0.0096 s. The accuracy of the six classifications was 0.8-0.9, and the accuracy of the two classifications was 0.75-0.8. The overall AUC values for each classification exceeded 0.9.

CONCLUSIONS

An automatic diagnostic classification model for lateral cephalograms was established based on multitask learning to achieve simultaneous classification of eight common clinical diagnostic items. The multitask learning model achieved better classification performance and reduced the computational costs, providing a novel perspective and reference for addressing such problems.

Longitudinal Image Data for Outcome Modeling

In oncology, medical imaging is crucial for diagnosis, treatment planning and therapy execution. Treatment responses can be complex and varied and are known to involve factors of treatment, patient characteristics and tumor microenvironment. Longitudinal image analysis is able to track temporal changes, aiding in disease monitoring, treatment evaluation, and outcome prediction. This allows for the enhancement of personalized medicine. However, analyzing longitudinal 2D and 3D images presents unique challenges, including image registration, reliable segmentation, dealing with variable imaging intervals, and sparse data. This review presents an overview of techniques and methodologies in longitudinal image analysis, with a primary focus on outcome modeling in radiation oncology.

Unsupervised deep learning-based medical image registration: a survey

In recent decades, medical image registration technology has undergone significant development, becoming one of the core technologies in medical image analysis. With the rise of deep learning, deep learning-based medical image registration methods have achieved revolutionary improvements in processing speed and automation, showing great potential, especially in unsupervised learning. This paper briefly introduces the core concepts of deep learning-based unsupervised image registration, followed by an in-depth discussion of innovative network architectures and a detailed review of these studies, highlighting their unique contributions. Additionally, this paper explores commonly used loss functions, datasets, and evaluation metrics. Finally, we discuss the main challenges faced by various categories and propose potential future research topics. This paper surveys the latest advancements in unsupervised deep neural network-based medical image registration methods, aiming to help active readers interested in this field gain a deep understanding of this exciting area.

A Multi-task learning U-Net model for end-to-end HEp-2 cell image analysis

Antinuclear Antibody (ANA) testing is pivotal to help diagnose patients with a suspected autoimmune disease. The Indirect Immunofluorescence (IIF) microscopy performed with human epithelial type 2 (HEp-2) cells as the substrate is the reference method for ANA screening. It allows for the detection of antibodies binding to specific intracellular targets, resulting in various staining patterns that should be identified for diagnosis purposes. In recent years, there has been an increasing interest in devising deep learning methods for automated cell segmentation and classification of staining patterns, as well as for other tasks related to this diagnostic technique (such as intensity classification). However, little attention has been devoted to architectures aimed at simultaneously managing multiple interrelated tasks, via a shared representation. In this paper, we propose a deep neural network model that extends U-Net in a Multi-Task Learning (MTL) fashion, thus offering an end-to-end approach to tackle three fundamental tasks of the diagnostic procedure, i.e., HEp-2 cell specimen intensity classification, specimen segmentation, and pattern classification. The experiments were conducted on one of the largest publicly available datasets of HEp-2 images. The results showed that the proposed approach significantly outperformed the competing state-of-the-art methods for all the considered tasks.

Multimodal data-based human motion intention prediction using adaptive hybrid deep learning network for movement challenged person

Recently, social demands for a good quality of life have increased among the elderly and disabled people. So, biomedical engineers and robotic researchers aimed to fuse these techniques in a novel rehabilitation system. Moreover, these models utilized the biomedical signals acquired from the human body's particular organ, cells, or tissues. The human motion intention prediction mechanism plays an essential role in various applications, such as assistive and rehabilitation robots, that execute specific tasks among elders and physically impaired individuals. However, more complications are increased in the human-machine-based interaction techniques, creating more scope for personalized assistance for the human motion intention prediction system. Therefore, in this paper, an Adaptive Hybrid Network (AHN) is implemented for effective human motion intention prediction. Initially, multimodal data like electroencephalogram (EEG)/Electromyography (EMG) signals and sensor measures data are collected from the available data resource. The gathered EEG/EMG signals are then converted into spectrogram images and sent to AH-CNN-LSTM, which is the integration of an Adaptive Hybrid Convolution Neural Network (AH-CNN) with a Long Short-Term Memory (LSTM) network. Similarly, the data details of sensor measures are directly subjected to AH-CNN-Res-LSTM, which is the combination of Adaptive Hybrid CNN with Residual Network and LSTM (Res-LSTM) to get the predictive result. Further, to enhance the prediction, the parameters in both the AH-CNN-LSTM and AH-CNN-Res-LSTM techniques are optimized using the Improved Yellow Saddle Goatfish Algorithm (IYSGA). The efficiency of the implemented model is computed by conducting the comparison experiment of the proposed technique with other standard models. The performance outcome of the developed method outperformed the other traditional methods.

Automated extraction of body composition metrics from abdominal CT or MR imaging: A scoping review

PURPOSE

To review methodological approaches for automated segmentation of subcutaneous adipose tissue, visceral adipose tissue, and skeletal muscle from abdominal cross-sectional imaging for body composition analysis.

METHOD

Four databases were searched for publications describing automated segmentation of subcutaneous adipose tissue, visceral adipose tissue, and/or skeletal muscle from abdominal CT or MR imaging between 2019 and 2023. Included reports were evaluated to assess how imaging modality, cohort size, vertebral level, model dimensionality, and use of a volume or single slice affected segmentation accuracy and/or clinical utility. Exclusion criteria included reports not in English language, manual or semi-automated segmentation methods, reports prior to 2019 or solely of paediatric patients, and those not describing the use of abdominal CT or MR.

RESULTS

After exclusions, 172 reports were included in the review. CT imaging was utilised approximately four times as often as MRI, and segmentation accuracy did not significantly differ between the two modalities. Cohort size had no significant effect on segmentation accuracy. There was little evidence to refute the current practice of extracting body composition metrics from the third lumbar vertebral level. There was no clear benefit of using a 3D model to perform segmentation over a 2D approach.

CONCLUSION

Automated segmentation of intra-abdominal soft tissues for body composition analysis is an intense area of research activity. Segmentation accuracy is not affected by cross-sectional imaging modality. Extracting metrics from a single slice at the third lumbar vertebral level is a common approach, however, extracting metrics from a volumetric slab surrounding this level may increase the resilience of the technique, which is important for clinical translation. A paucity of publicly available datasets led to most reports using different data sources, preventing direct comparison of segmentation techniques. Future efforts should prioritise creating a standardised dataset to facilitate benchmarking of different algorithms and subsequent clinical adoption.

A Unified Multi-Task Learning Model with Joint Reverse Optimization for Simultaneous Skin Lesion Segmentation and Diagnosis

Classifying and segmenting skin cancer represent pivotal objectives for automated diagnostic systems that utilize dermoscopy images. However, these tasks present significant challenges due to the diverse shape variations of skin lesions and the inherently fuzzy nature of dermoscopy images, including low contrast and the presence of artifacts. Given the robust correlation between the classification of skin lesions and their segmentation, we propose that employing a combined learning method holds the promise of considerably enhancing the performance of both tasks. In this paper, we present a unified multi-task learning strategy that concurrently classifies abnormalities of skin lesions and allows for the joint segmentation of lesion boundaries. This approach integrates an optimization technique known as joint reverse learning, which fosters mutual enhancement through extracting shared features and limiting task dominance across the two tasks. The effectiveness of the proposed method was assessed using two publicly available datasets, ISIC 2016 and PH2, which included melanoma and benign skin cancers. In contrast to the single-task learning strategy, which solely focuses on either classification or segmentation, the experimental findings demonstrated that the proposed network improves the diagnostic capability of skin tumor screening and analysis. The proposed method achieves a significant segmentation performance on skin lesion boundaries, with Dice Similarity Coefficients (DSC) of 89.48% and 88.81% on the ISIC 2016 and PH2 datasets, respectively. Additionally, our multi-task learning approach enhances classification, increasing the F1 score from 78.26% (baseline ResNet50) to 82.07% on ISIC 2016 and from 82.38% to 85.50% on PH2. This work showcases its potential applicability across varied clinical scenarios.

Integrating multi-task and cost-sensitive learning for predicting mortality risk of chronic diseases in the elderly using real-world data

BACKGROUND AND OBJECTIVE

Real-world data encompass population diversity, enabling insights into chronic disease mortality risk among the elderly. Deep learning excels on large datasets, offering promise for real-world data. However, current models focus on single diseases, neglecting comorbidities prevalent in patients. Moreover, mortality is infrequent compared to illness, causing extreme class imbalance that impedes reliable prediction. We aim to develop a deep learning framework that accurately forecasts mortality risk from real-world data by addressing comorbidities and class imbalance.

METHODS

We integrated multi-task and cost-sensitive learning, developing an enhanced deep neural network architecture that extends multi-task learning to predict mortality risk across multiple chronic diseases. Each patient cohort with a chronic disease was assigned to a separate task, with shared lower-level parameters capturing inter-disease complexities through distinct top-level networks. Cost-sensitive functions were incorporated to ensure learning of positive class characteristics for each task and achieve accurate prediction of the risk of death from multiple chronic diseases.

RESULTS

Our study covers 15 prevalent chronic diseases and is experimented with real-world data from 482,145 patients (including 9,516 deaths) in Shenzhen, China. The proposed model is compared with six models including three machine learning models: logistic regression, XGBoost, and CatBoost, and three state-of-the-art deep learning models: 1D-CNN, TabNet, and Saint. The experimental results show that, compared with the other compared algorithms, MTL-CSDNN has better prediction results on the test set (ACC=0.99, REC=0.99, PRAUC=0.97, MCC=0.98, G-means = 0.98).

CONCLUSIONS

Our method provides valuable insights into leveraging real-world data for precise multi-disease mortality risk prediction, offering potential applications in optimizing chronic disease management, enhancing well-being, and reducing healthcare costs for the elderly population.

Adaptive segmentation-to-survival learning for survival prediction from multi-modality medical images

Early survival prediction is vital for the clinical management of cancer patients, as tumors can be better controlled with personalized treatment planning. Traditional survival prediction methods are based on radiomics feature engineering and/or clinical indicators (e.g., cancer staging). Recently, survival prediction models with advances in deep learning techniques have achieved state-of-the-art performance in end-to-end survival prediction by exploiting deep features derived from medical images. However, existing models are heavily reliant on the prognostic information within primary tumors and cannot effectively leverage out-of-tumor prognostic information characterizing local tumor metastasis and adjacent tissue invasion. Also, existing models are sub-optimal in leveraging multi-modality medical images as they rely on empirically designed fusion strategies to integrate multi-modality information, where the fusion strategies are pre-defined based on domain-specific human prior knowledge and inherently limited in adaptability. Here, we present an Adaptive Multi-modality Segmentation-to-Survival model (AdaMSS) for survival prediction from multi-modality medical images. The AdaMSS can self-adapt its fusion strategy based on training data and also can adapt its focus regions to capture the prognostic information outside the primary tumors. Extensive experiments with two large cancer datasets (1380 patients from nine medical centers) show that our AdaMSS surmounts the state-of-the-art survival prediction performance (C-index: 0.804 and 0.757), demonstrating the potential to facilitate personalized treatment planning.

Deep Convolutional Neural Network for Dedicated Regions-of-Interest Based Multi-Parameter Quantitative Ultrashort Echo Time (UTE) Magnetic Resonance Imaging of the Knee Joint

We proposed an end-to-end deep learning convolutional neural network (DCNN) for region-of-interest based multi-parameter quantification (RMQ-Net) to accelerate quantitative ultrashort echo time (UTE) MRI of the knee joint with automatic multi-tissue segmentation and relaxometry mapping. The study involved UTE-based T1 (UTE-T1) and Adiabatic T1ρ (UTE-AdiabT1ρ) mapping of the knee joint of 65 human subjects, including 20 normal controls, 29 with doubtful-minimal osteoarthritis (OA), and 16 with moderate-severe OA. Comparison studies were performed on UTE-T1 and UTE-AdiabT1ρ measurements using 100%, 43%, 26%, and 18% UTE MRI data as the inputs and the effects on the prediction quality of the RMQ-Net. The RMQ-net was modified and retrained accordingly with different combinations of inputs. Both ROI-based and voxel-based Pearson correlation analyses were performed. High Pearson correlation coefficients were achieved between the RMQ-Net predicted UTE-T1 and UTE-AdiabT1ρ results and the ground truth for segmented cartilage with acceleration factors ranging from 2.3 to 5.7. With an acceleration factor of 5.7, the Pearson r-value achieved 0.908 (ROI-based) and 0.945 (voxel-based) for UTE-T1, and 0.733 (ROI-based) and 0.895 (voxel-based) for UTE-AdiabT1ρ, correspondingly. The results demonstrated that RMQ-net can significantly accelerate quantitative UTE imaging with automated segmentation of articular cartilage in the knee joint.

Machine learning approaches to detect hepatocyte chromatin alterations from iron oxide nanoparticle exposure

This study focuses on developing machine learning models to detect subtle alterations in hepatocyte chromatin organization due to Iron (II, III) oxide nanoparticle exposure, hypothesizing that exposure will significantly alter chromatin texture. A total of 2000 hepatocyte nuclear regions of interest (ROIs) from mouse liver tissue were analyzed, and for each ROI, 5 different parameters were calculated: Long Run Emphasis, Short Run Emphasis, Run Length Nonuniformity, and 2 wavelet coefficient energies obtained after the discrete wavelet transform. These parameters served as input for supervised machine learning models, specifically random forest and gradient boosting classifiers. The models demonstrated relatively robust performance in distinguishing hepatocyte chromatin structures belonging to the group exposed to IONPs from the controls. The study's findings suggest that iron oxide nanoparticles induce substantial changes in hepatocyte chromatin distribution and underscore the potential of AI techniques in advancing hepatocyte evaluation in physiological and pathological conditions.

Real-time estimation of the optimal coil placement in transcranial magnetic stimulation using multi-task deep learning

Transcranial magnetic stimulation (TMS) has emerged as a promising neuromodulation technique with both therapeutic and diagnostic applications. As accurate coil placement is known to be essential for focal stimulation, computational models have been established to help find the optimal coil positioning by maximizing electric fields at the cortical target. While these numerical simulations provide realistic and subject-specific field distributions, they are computationally demanding, precluding their use in real-time applications. In this paper, we developed a novel multi-task deep neural network which simultaneously predicts the optimal coil placement for a given cortical target as well as the associated TMS-induced electric field. Trained on large amounts of preceding numerical optimizations, the Attention U-Net-based neural surrogate provided accurate coil optimizations in only 35 ms, a fraction of time compared to the state-of-the-art numerical framework. The mean errors on the position estimates were below 2 mm, i.e., smaller than previously reported manual coil positioning errors. The predicted electric fields were also highly correlated (r> 0.97) with their numerical references. In addition to healthy subjects, we validated our approach also in glioblastoma patients. We first statistically underlined the importance of using realistic heterogeneous tumor conductivities instead of simply adopting values from the surrounding healthy tissue. Second, applying the trained neural surrogate to tumor patients yielded similar accurate positioning and electric field estimates as in healthy subjects. Our findings provide a promising framework for future real-time electric field-optimized TMS applications.

Biologically interpretable multi-task deep learning pipeline predicts molecular alterations, grade, and prognosis in glioma patients

Deep learning models have been developed for various predictions in glioma; yet, they were constrained by manual segmentation, task-specific design, or a lack of biological interpretation. Herein, we aimed to develop an end-to-end multi-task deep learning (MDL) pipeline that can simultaneously predict molecular alterations and histological grade (auxiliary tasks), as well as prognosis (primary task) in gliomas. Further, we aimed to provide the biological mechanisms underlying the model's predictions. We collected multiscale data including baseline MRI images from 2776 glioma patients across two private (FAHZU and HPPH, n = 1931) and three public datasets (TCGA, n = 213; UCSF, n = 410; and EGD, n = 222). We trained and internally validated the MDL model using our private datasets, and externally validated it using the three public datasets. We used the model-predicted deep prognosis score (DPS) to stratify patients into low-DPS and high-DPS subtypes. Additionally, a radio-multiomics analysis was conducted to elucidate the biological basis of the DPS. In the external validation cohorts, the MDL model achieved average areas under the curve of 0.892-0.903, 0.710-0.894, and 0.850-0.879 for predicting IDH mutation status, 1p/19q co-deletion status, and tumor grade, respectively. Moreover, the MDL model yielded a C-index of 0.723 in the TCGA and 0.671 in the UCSF for the prediction of overall survival. The DPS exhibits significant correlations with activated oncogenic pathways, immune infiltration patterns, specific protein expression, DNA methylation, tumor mutation burden, and tumor-stroma ratio. Accordingly, our work presents an accurate and biologically meaningful tool for predicting molecular subtypes, tumor grade, and survival outcomes in gliomas, which provides personalized clinical decision-making in a global and non-invasive manner.

Knowledge‐based deep learning system for classifying Alzheimer's disease for multi‐task learning

Deep learning has recently become a viable approach for classifying Alzheimer's disease (AD) in medical imaging. However, existing models struggle to efficiently extract features from medical images and may squander additional information resources for illness classification. To address these issues, a deep three‐dimensional convolutional neural network incorporating multi‐task learning and attention mechanisms is proposed. An upgraded primary C3D network is utilised to create rougher low‐level feature maps. It introduces a new convolution block that focuses on the structural aspects of the magnetic resonance imaging image and another block that extracts attention weights unique to certain pixel positions in the feature map and multiplies them with the feature map output. Then, several fully connected layers are used to achieve multi‐task learning, generating three outputs, including the primary classification task. The other two outputs employ backpropagation during training to improve the primary classification job. Experimental findings show that the authors’ proposed method outperforms current approaches for classifying AD, achieving enhanced classification accuracy and other indicators on the Alzheimer's disease Neuroimaging Initiative dataset. The authors demonstrate promise for future disease classification studies.

MPEK: a multitask deep learning framework based on pretrained language models for enzymatic reaction kinetic parameters prediction

Enzymatic reaction kinetics are central in analyzing enzymatic reaction mechanisms and target-enzyme optimization, and thus in biomanufacturing and other industries. The enzyme turnover number (kcat) and Michaelis constant (Km), key kinetic parameters for measuring enzyme catalytic efficiency, are crucial for analyzing enzymatic reaction mechanisms and the directed evolution of target enzymes. Experimental determination of kcat and Km is costly in terms of time, labor, and cost. To consider the intrinsic connection between kcat and Km and further improve the prediction performance, we propose a universal pretrained multitask deep learning model, MPEK, to predict these parameters simultaneously while considering pH, temperature, and organismal information. Through testing on the same kcat and Km test datasets, MPEK demonstrated superior prediction performance over the previous models. Specifically, MPEK achieved the Pearson coefficient of 0.808 for predicting kcat, improving ca. 14.6% and 7.6% compared to the DLKcat and UniKP models, and it achieved the Pearson coefficient of 0.777 for predicting Km, improving ca. 34.9% and 53.3% compared to the Kroll_model and UniKP models. More importantly, MPEK was able to reveal enzyme promiscuity and was sensitive to slight changes in the mutant enzyme sequence. In addition, in three case studies, it was shown that MPEK has the potential for assisted enzyme mining and directed evolution. To facilitate in silico evaluation of enzyme catalytic efficiency, we have established a web server implementing this model, which can be accessed at http://mathtc.nscc-tj.cn/mpek.

Bi-Fusion of Structure and Deformation at Multi-Scale for Joint Segmentation and Registration

Medical image segmentation and registration are two fundamental and highly related tasks. However, current works focus on the mutual promotion between the two at the loss function level, ignoring the feature information generated by the encoder-decoder network during the task-specific feature mapping process and the potential inter-task feature relationship. This paper proposes a unified multi-task joint learning framework based on bi-fusion of structure and deformation at multi-scale, called BFM-Net, which simultaneously achieves the segmentation results and deformation field in a single-step estimation. BFM-Net consists of a segmentation subnetwork (SegNet), a registration subnetwork (RegNet), and the multi-task connection module (MTC). The MTC module is used to transfer the latent feature representation between segmentation and registration at multi-scale and link different tasks at the network architecture level, including the spatial attention fusion module (SAF), the multi-scale spatial attention fusion module (MSAF) and the velocity field fusion module (VFF). Extensive experiments on MR, CT and ultrasound images demonstrate the effectiveness of our approach. The MTC module can increase the Dice scores of segmentation and registration by 3.2%, 1.6%, 2.2%, and 6.2%, 4.5%, 3.0%, respectively. Compared with six state-of-the-art algorithms for segmentation and registration, BFM-Net can achieve superior performance in various modal images, fully demonstrating its effectiveness and generalization.

Deep Learning for Automated Detection and Localization of Traumatic Abdominal Solid Organ Injuries on CT Scans

Computed tomography (CT) is the most commonly used diagnostic modality for blunt abdominal trauma (BAT), significantly influencing management approaches. Deep learning models (DLMs) have shown great promise in enhancing various aspects of clinical practice. There is limited literature available on the use of DLMs specifically for trauma image evaluation. In this study, we developed a DLM aimed at detecting solid organ injuries to assist medical professionals in rapidly identifying life-threatening injuries. The study enrolled patients from a single trauma center who received abdominal CT scans between 2008 and 2017. Patients with spleen, liver, or kidney injury were categorized as the solid organ injury group, while others were considered negative cases. Only images acquired from the trauma center were enrolled. A subset of images acquired in the last year was designated as the test set, and the remaining images were utilized to train and validate the detection models. The performance of each model was assessed using metrics such as the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, positive predictive value, and negative predictive value based on the best Youden index operating point. The study developed the models using 1302 (87%) scans for training and tested them on 194 (13%) scans. The spleen injury model demonstrated an accuracy of 0.938 and a specificity of 0.952. The accuracy and specificity of the liver injury model were reported as 0.820 and 0.847, respectively. The kidney injury model showed an accuracy of 0.959 and a specificity of 0.989. We developed a DLM that can automate the detection of solid organ injuries by abdominal CT scans with acceptable diagnostic accuracy. It cannot replace the role of clinicians, but we can expect it to be a potential tool to accelerate the process of therapeutic decisions for trauma care.

Accurate object localization facilitates automatic esophagus segmentation in deep learning

BACKGROUND

Currently, automatic esophagus segmentation remains a challenging task due to its small size, low contrast, and large shape variation. We aimed to improve the performance of esophagus segmentation in deep learning by applying a strategy that involves locating the object first and then performing the segmentation task.

METHODS

A total of 100 cases with thoracic computed tomography scans from two publicly available datasets were used in this study. A modified CenterNet, an object location network, was employed to locate the center of the esophagus for each slice. Subsequently, the 3D U-net and 2D U-net_coarse models were trained to segment the esophagus based on the predicted object center. A 2D U-net_fine model was trained based on the updated object center according to the 3D U-net model. The dice similarity coefficient and the 95% Hausdorff distance were used as quantitative evaluation indexes for the delineation performance. The characteristics of the automatically delineated esophageal contours by the 2D U-net and 3D U-net models were summarized. Additionally, the impact of the accuracy of object localization on the delineation performance was analyzed. Finally, the delineation performance in different segments of the esophagus was also summarized.

RESULTS

The mean dice coefficient of the 3D U-net, 2D U-net_coarse, and 2D U-net_fine models were 0.77, 0.81, and 0.82, respectively. The 95% Hausdorff distance for the above models was 6.55, 3.57, and 3.76, respectively. Compared with the 2D U-net, the 3D U-net has a lower incidence of delineating wrong objects and a higher incidence of missing objects. After using the fine object center, the average dice coefficient was improved by 5.5% in the cases with a dice coefficient less than 0.75, while that value was only 0.3% in the cases with a dice coefficient greater than 0.75. The dice coefficients were lower for the esophagus between the orifice of the inferior and the pulmonary bifurcation compared with the other regions.

CONCLUSION

The 3D U-net model tended to delineate fewer incorrect objects but also miss more objects. Two-stage strategy with accurate object location could enhance the robustness of the segmentation model and significantly improve the esophageal delineation performance, especially for cases with poor delineation results.

Fruit freshness detection based on multi-task convolutional neural network

Background

Fruit freshness detection by computer vision is essential for many agricultural applications, e.g., automatic harvesting and supply chain monitoring. This paper proposes to use the multi-task learning (MTL) paradigm to build a deep convolutional neural work for fruit freshness detection.

Results

We design an MTL model that optimizes the freshness detection (T1) and fruit type classification (T2) tasks in parallel. The model uses a shared CNN (convolutional neural network) subnet and two FC (fully connected) task heads. The shared CNN acts as a feature extraction module and feeds the two task heads with common semantic features. Based on an open fruit image dataset, we conducted a comparative study of MTL and single-task learning (STL) paradigms. The STL models use the same CNN subnet with only one specific task head. In the MTL scenario, the T1 and T2 mean accuracies on the test set are 93.24% and 88.66%, respectively. Meanwhile, for STL, the two accuracies are 92.50% and 87.22%. Statistical tests report significant differences between MTL and STL on T1 and T2 test accuracies. We further investigated the extracted feature vectors (semantic embeddings) from the two STL models. The vectors have an averaged 0.7 cosine similarity on the entire dataset, with most values lying in the 0.6-0.8 range. This indicates a between-task correlation and justifies the effectiveness of the proposed MTL approach.

Conclusion

This study proves that MTL exploits the mutual correlation between two or more relevant tasks and can maximally share their underlying feature extraction process. we envision this approach to be extended to other domains that involve multiple interconnected tasks.

Brain tumor segmentation using neuro-technology enabled intelligence-cascaded U-Net model

According to experts in neurology, brain tumours pose a serious risk to human health. The clinical identification and treatment of brain tumours rely heavily on accurate segmentation. The varied sizes, forms, and locations of brain tumours make accurate automated segmentation a formidable obstacle in the field of neuroscience. U-Net, with its computational intelligence and concise design, has lately been the go-to model for fixing medical picture segmentation issues. Problems with restricted local receptive fields, lost spatial information, and inadequate contextual information are still plaguing artificial intelligence. A convolutional neural network (CNN) and a Mel-spectrogram are the basis of this cough recognition technique. First, we combine the voice in a variety of intricate settings and improve the audio data. After that, we preprocess the data to make sure its length is consistent and create a Mel-spectrogram out of it. A novel model for brain tumor segmentation (BTS), Intelligence Cascade U-Net (ICU-Net), is proposed to address these issues. It is built on dynamic convolution and uses a non-local attention mechanism. In order to reconstruct more detailed spatial information on brain tumours, the principal design is a two-stage cascade of 3DU-Net. The paper's objective is to identify the best learnable parameters that will maximize the likelihood of the data. After the network's ability to gather long-distance dependencies for AI, Expectation-Maximization is applied to the cascade network's lateral connections, enabling it to leverage contextual data more effectively. Lastly, to enhance the network's ability to capture local characteristics, dynamic convolutions with local adaptive capabilities are used in place of the cascade network's standard convolutions. We compared our results to those of other typical methods and ran extensive testing utilising the publicly available BraTS 2019/2020 datasets. The suggested method performs well on tasks involving BTS, according to the experimental data. The Dice scores for tumor core (TC), complete tumor, and enhanced tumor segmentation BraTS 2019/2020 validation sets are 0.897/0.903, 0.826/0.828, and 0.781/0.786, respectively, indicating high performance in BTS.

Rogue AI: Cautionary Cases in Neuroradiology and What We Can Learn From Them

Introduction In recent years, artificial intelligence (AI) in medical imaging has undergone unprecedented innovation and advancement, sparking a revolutionary transformation in healthcare. The field of radiology is particularly implicated, as clinical radiologists are expected to interpret an ever-increasing number of complex cases in record time. Machine learning software purchased by our institution is expected to help our radiologists come to a more prompt diagnosis by delivering point-of-care quantitative analysis of suspicious findings and streamlining clinical workflow. This paper explores AI's impact on neuroradiology, an area accounting for a substantial portion of recent radiology studies. We present a case series evaluating an AI software's performance in detecting neurovascular findings, highlighting five cases where AI interpretations differed from radiologists' assessments. Our study underscores common pitfalls of AI in the context of CT head angiograms, aiming to guide future AI algorithms.  Methods We conducted a retrospective case series study at Stony Brook University Hospital, a large medical center in Stony Brook, New York, spanning from October 1, 2021 to December 31, 2021, analyzing 140 randomly sampled CT angiograms using AI software. This software assessed various neurovascular parameters, and AI findings were compared with neuroradiologists' interpretations. Five cases with divergent interpretations were selected for detailed analysis. Results Five representative cases in which AI findings were discordant with radiologists' interpretations are presented with diagnoses including diffuse anoxic ischemic injury, cortical laminar necrosis, colloid cyst, right superficial temporal artery-to-middle cerebral artery (STA-MCA) bypass, and subacute bilateral subdural hematomas. Discussion The errors identified in our case series expose AI's limitations in radiology. Our case series reveals that AI's incorrect interpretations can stem from complexities in pathology, challenges in distinguishing densities, inability to identify artifacts, identifying post-surgical changes in normal anatomy, sensitivity limitations, and insufficient pattern recognition. AI's potential for improvement lies in refining its algorithms to effectively recognize and differentiate pathologies. Incorporating more diverse training datasets, multimodal data, deep-reinforcement learning, clinical context, and real-time learning capabilities are some ways to improve AI's performance in the field of radiology. Conclusion Overall, it is apparent that AI applications in radiology have much room for improvement before becoming more widely integrated into clinical workflows. While AI demonstrates remarkable potential to aid in diagnosis and streamline workflows, our case series highlights common pitfalls that underscore the need for continuous improvement. By refining algorithms, incorporating diverse datasets, embracing multimodal information, and leveraging innovative machine learning strategies, AI's diagnostic accuracy can be significantly improved.

Trends and Hotspots in Global Radiomics Research: A Bibliometric Analysis

Objectives: The purpose of this research is to summarize the structure of radiomics-based knowledge and to explore potential trends and priorities by using bibliometric analysis. Methods: Select radiomics-related publications from 2012 to October 2022 from the Science Core Collection Web site. Use VOSviewer (version 1.6.18), CiteSpace (version 6.1.3), Tableau (version 2022), Microsoft Excel and Rstudio's free online platforms (http://bibliometric.com) for co-writing, co-citing, and co-occurrence analysis of countries, institutions, authors, references, and keywords in the field. The visual analysis is also carried out on it. Results: The study included 6428 articles. Since 2012, there has been an increase in research papers based on radiomics. Judging by publications, China has made the largest contribution in this area. We identify the most productive institutions and authors as Fudan University and Tianjie. The top three magazines with the most publications are《FRONTIERS IN ONCOLOGY》, 《EUROPEAN RADIOLOGY》, and 《CANCERS》. According to the results of reference and keyword analysis, "deep learning, nomogram, ultrasound, f-18-fdg, machine learning, covid-19, radiogenomics" has been determined as the main research direction in the future. Conclusion: Radiomics is in a phase of vigorous development with broad prospects. Cross-border cooperation between countries and institutions should be strengthened in the future. It can be predicted that the development of deep learning-based models and multimodal fusion models will be the focus of future research. Advances in knowledge: This study explores the current state of research and hot spots in the field of radiomics from multiple perspectives, comprehensively, and objectively reflecting the evolving trends in imaging-related research and providing a reference for future research.

Revolutionizing tumor detection and classification in multimodality imaging based on deep learning approaches: Methods, applications and limitations

BACKGROUND

The emergence of deep learning (DL) techniques has revolutionized tumor detection and classification in medical imaging, with multimodal medical imaging (MMI) gaining recognition for its precision in diagnosis, treatment, and progression tracking.

OBJECTIVE

This review comprehensively examines DL methods in transforming tumor detection and classification across MMI modalities, aiming to provide insights into advancements, limitations, and key challenges for further progress.

METHODS

Systematic literature analysis identifies DL studies for tumor detection and classification, outlining methodologies including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their variants. Integration of multimodality imaging enhances accuracy and robustness.

RESULTS

Recent advancements in DL-based MMI evaluation methods are surveyed, focusing on tumor detection and classification tasks. Various DL approaches, including CNNs, YOLO, Siamese Networks, Fusion-Based Models, Attention-Based Models, and Generative Adversarial Networks, are discussed with emphasis on PET-MRI, PET-CT, and SPECT-CT.

FUTURE DIRECTIONS

The review outlines emerging trends and future directions in DL-based tumor analysis, aiming to guide researchers and clinicians toward more effective diagnosis and prognosis. Continued innovation and collaboration are stressed in this rapidly evolving domain.

CONCLUSION

Conclusions drawn from literature analysis underscore the efficacy of DL approaches in tumor detection and classification, highlighting their potential to address challenges in MMI analysis and their implications for clinical practice.

Updated Models of Alzheimer's Disease with Deep Neural Networks

Background

In recent years, researchers have focused on developing precise models for the progression of Alzheimer's disease (AD) using deep neural networks. Forecasting the progression of AD through the analysis of time series data represents a promising approach.

Objective

The primary objective of this research is to formulate an effective methodology for forecasting the progression of AD through the integration of multi-task learning techniques and the analysis of pertinent medical data.

Methods

This study primarily utilized volumetric measurements obtained through magnetic resonance imaging (MRI), trajectories of cognitive assessments, and clinical status indicators. The research encompassed 150 patients diagnosed with AD who underwent examination between 2020 and 2022 in Beijing, China. A multi-task learning approach was employed to train forecasting models using MRI data, trajectories of cognitive assessments, and clinical status. Correlation analysis was conducted at various time points.

Results

At the baseline, a robust correlation was observed among the forecasting tasks: 0.75 for volumetric MRI measurements, 0.62 for trajectories of cognitive assessment, and 0.48 for clinical status. The implementation of a multi-task learning framework enhanced performance by 12.7% for imputing missing values and 14.8% for prediction accuracy.

Conclusions

The findings of our study, indicate that multi-task learning can effectively predict the progression of AD. However, it is important to note that the study's generalizability may be limited due to the restricted dataset and the specific population under examination. These conclusions represent a significant stride toward more precise diagnosis and treatment of this neurological disorder.

Survey of Transfer Learning Approaches in the Machine Learning of Digital Health Sensing Data

Machine learning and digital health sensing data have led to numerous research achievements aimed at improving digital health technology. However, using machine learning in digital health poses challenges related to data availability, such as incomplete, unstructured, and fragmented data, as well as issues related to data privacy, security, and data format standardization. Furthermore, there is a risk of bias and discrimination in machine learning models. Thus, developing an accurate prediction model from scratch can be an expensive and complicated task that often requires extensive experiments and complex computations. Transfer learning methods have emerged as a feasible solution to address these issues by transferring knowledge from a previously trained task to develop high-performance prediction models for a new task. This survey paper provides a comprehensive study of the effectiveness of transfer learning for digital health applications to enhance the accuracy and efficiency of diagnoses and prognoses, as well as to improve healthcare services. The first part of this survey paper presents and discusses the most common digital health sensing technologies as valuable data resources for machine learning applications, including transfer learning. The second part discusses the meaning of transfer learning, clarifying the categories and types of knowledge transfer. It also explains transfer learning methods and strategies, and their role in addressing the challenges in developing accurate machine learning models, specifically on digital health sensing data. These methods include feature extraction, fine-tuning, domain adaptation, multitask learning, federated learning, and few-/single-/zero-shot learning. This survey paper highlights the key features of each transfer learning method and strategy, and discusses the limitations and challenges of using transfer learning for digital health applications. Overall, this paper is a comprehensive survey of transfer learning methods on digital health sensing data which aims to inspire researchers to gain knowledge of transfer learning approaches and their applications in digital health, enhance the current transfer learning approaches in digital health, develop new transfer learning strategies to overcome the current limitations, and apply them to a variety of digital health technologies.

Towards fully automated inner ear analysis with deep-learning-based joint segmentation and landmark detection framework

Automated analysis of the inner ear anatomy in radiological data instead of time-consuming manual assessment is a worthwhile goal that could facilitate preoperative planning and clinical research. We propose a framework encompassing joint semantic segmentation of the inner ear and anatomical landmark detection of helicotrema, oval and round window. A fully automated pipeline with a single, dual-headed volumetric 3D U-Net was implemented, trained and evaluated using manually labeled in-house datasets from cadaveric specimen ([Formula: see text]) and clinical practice ([Formula: see text]). The model robustness was further evaluated on three independent open-source datasets ([Formula: see text] scans) consisting of cadaveric specimen scans. For the in-house datasets, Dice scores of [Formula: see text], intersection-over-union scores of [Formula: see text] and average Hausdorff distances of [Formula: see text] and [Formula: see text] voxel units were achieved. The landmark localization task was performed automatically with an average localization error of [Formula: see text] voxel units. A robust, albeit reduced performance could be attained for the catalogue of three open-source datasets. Results of the ablation studies with 43 mono-parametric variations of the basal architecture and training protocol provided task-optimal parameters for both categories. Ablation studies against single-task variants of the basal architecture showed a clear performance benefit of coupling landmark localization with segmentation and a dataset-dependent performance impact on segmentation ability.

Enhancing Fetal Anomaly Detection in Ultrasonography Images: A Review of Machine Learning-Based Approaches

Fetal development is a critical phase in prenatal care, demanding the timely identification of anomalies in ultrasound images to safeguard the well-being of both the unborn child and the mother. Medical imaging has played a pivotal role in detecting fetal abnormalities and malformations. However, despite significant advances in ultrasound technology, the accurate identification of irregularities in prenatal images continues to pose considerable challenges, often necessitating substantial time and expertise from medical professionals. In this review, we go through recent developments in machine learning (ML) methods applied to fetal ultrasound images. Specifically, we focus on a range of ML algorithms employed in the context of fetal ultrasound, encompassing tasks such as image classification, object recognition, and segmentation. We highlight how these innovative approaches can enhance ultrasound-based fetal anomaly detection and provide insights for future research and clinical implementations. Furthermore, we emphasize the need for further research in this domain where future investigations can contribute to more effective ultrasound-based fetal anomaly detection.