Automated Muscle Segmentation from Clinical CT Using Bayesian U-Net for Personalized Musculoskeletal Modeling

A Bayesian network for simultaneous keyframe and landmark detection in ultrasonic cine

Accurate landmark detection in medical imaging is essential for quantifying various anatomical structures and assisting in diagnosis and treatment planning. In ultrasound cine, landmark detection is often associated with identifying keyframes, which represent the occurrence of specific events, such as measuring target dimensions at specific temporal phases. Existing methods predominantly treat landmark and keyframe detection as separate tasks without harnessing their underlying correlations. Additionally, owing to the intrinsic characteristics of ultrasound imaging, both tasks are constrained by inter-observer variability, leading to potentially higher levels of uncertainty. In this paper, we propose a Bayesian network to achieve simultaneous keyframe and landmark detection in ultrasonic cine, especially under highly sparse training data conditions. We follow a coarse-to-fine landmark detection architecture and propose an adaptive Bayesian hypergraph for coordinate refinement on the results of heatmap-based regression. In addition, we propose Order Loss for training bi-directional Gated Recurrent Unit to identify keyframes based on the relative likelihoods within the sequence. Furthermore, to exploit the underlying correlation between the two tasks, we use a shared encoder to extract features for both tasks and enhance the detection accuracy through the interaction of temporal and motion information. Experiments on two in-house datasets (multi-view transesophageal and transthoracic echocardiography) and one public dataset (transthoracic echocardiography) demonstrate that our method outperforms state-of-the-art approaches. The mean absolute errors for dimension measurements of the left atrial appendage, aortic annulus, and left ventricle are 2.40 mm, 0.83 mm, and 1.63 mm, respectively. The source code is available at github.com/warmestwind/ABHG.

Comprehensive review of deep learning in orthopaedics: Applications, challenges, trustworthiness, and fusion

Deep learning (DL) in orthopaedics has gained significant attention in recent years. Previous studies have shown that DL can be applied to a wide variety of orthopaedic tasks, including fracture detection, bone tumour diagnosis, implant recognition, and evaluation of osteoarthritis severity. The utilisation of DL is expected to increase, owing to its ability to present accurate diagnoses more efficiently than traditional methods in many scenarios. This reduces the time and cost of diagnosis for patients and orthopaedic surgeons. To our knowledge, no exclusive study has comprehensively reviewed all aspects of DL currently used in orthopaedic practice. This review addresses this knowledge gap using articles from Science Direct, Scopus, IEEE Xplore, and Web of Science between 2017 and 2023. The authors begin with the motivation for using DL in orthopaedics, including its ability to enhance diagnosis and treatment planning. The review then covers various applications of DL in orthopaedics, including fracture detection, detection of supraspinatus tears using MRI, osteoarthritis, prediction of types of arthroplasty implants, bone age assessment, and detection of joint-specific soft tissue disease. We also examine the challenges for implementing DL in orthopaedics, including the scarcity of data to train DL and the lack of interpretability, as well as possible solutions to these common pitfalls. Our work highlights the requirements to achieve trustworthiness in the outcomes generated by DL, including the need for accuracy, explainability, and fairness in the DL models. We pay particular attention to fusion techniques as one of the ways to increase trustworthiness, which have also been used to address the common multimodality in orthopaedics. Finally, we have reviewed the approval requirements set forth by the US Food and Drug Administration to enable the use of DL applications. As such, we aim to have this review function as a guide for researchers to develop a reliable DL application for orthopaedic tasks from scratch for use in the market.

Deep learning-based respiratory muscle segmentation as a potential imaging biomarker for respiratory function assessment

Respiratory diseases significantly affect respiratory function, making them a considerable contributor to global mortality. The respiratory muscles play an important role in disease prognosis; as such, quantitative analysis of the respiratory muscles is crucial to assess the status of the respiratory system and the quality of life in patients. In this study, we aimed to develop an automated approach for the segmentation and classification of three types of respiratory muscles from computed tomography (CT) images using artificial intelligence. With a dataset of approximately 600,000 thoracic CT images from 3,200 individuals, we trained the model using the Attention U-Net architecture, optimized for detailed and focused segmentation. Subsequently, we calculated the volumes and densities from the muscle masks segmented by our model and performed correlation analysis with pulmonary function test (PFT) parameters. The segmentation models for muscle tissue and respiratory muscles obtained dice scores of 0.9823 and 0.9688, respectively. The classification model, achieving a generalized dice score of 0.9900, also demonstrated high accuracy in classifying thoracic region muscle types, as evidenced by its F1 scores: 0.9793 for the pectoralis muscle, 0.9975 for the erector spinae muscle, and 0.9839 for the intercostal muscle. In the correlation analysis, the volume of the respiratory muscles showed a strong correlation with PFT parameters, suggesting that respiratory muscle volume may serve as a potential novel biomarker for respiratory function. Although muscle density showed a weaker correlation with the PFT parameters, it has a potential significance in medical research.

Efficient musculoskeletal annotation using free-form deformation

Traditionally, constructing training datasets for automatic muscle segmentation from medical images involved skilled operators, leading to high labor costs and limited scalability. To address this issue, we developed a tool that enables efficient annotation by non-experts and assessed its effectiveness for training an automatic segmentation network. Our system allows users to deform a template three-dimensional (3D) anatomical model to fit a target magnetic-resonance image using free-form deformation with independent control points for axial, sagittal, and coronal directions. This method simplifies the annotation process by allowing non-experts to intuitively adjust the model, enabling simultaneous annotation of all muscles in the template. We evaluated the quality of the tool-assisted segmentation performed by non-experts, which achieved a Dice coefficient greater than 0.75 compared to expert segmentation, without significant errors such as mislabeling adjacent muscles or omitting musculature. An automatic segmentation network trained with datasets created using this tool demonstrated performance comparable to or superior to that of networks trained with expert-generated datasets. This innovative tool significantly reduces the time and labor costs associated with dataset creation for automatic muscle segmentation, potentially revolutionizing medical image annotation and accelerating the development of deep learning-based segmentation networks in various clinical applications.

A comprehensive survey on deep active learning in medical image analysis

Deep learning has achieved widespread success in medical image analysis, leading to an increasing demand for large-scale expert-annotated medical image datasets. Yet, the high cost of annotating medical images severely hampers the development of deep learning in this field. To reduce annotation costs, active learning aims to select the most informative samples for annotation and train high-performance models with as few labeled samples as possible. In this survey, we review the core methods of active learning, including the evaluation of informativeness and sampling strategy. For the first time, we provide a detailed summary of the integration of active learning with other label-efficient techniques, such as semi-supervised, self-supervised learning, and so on. We also summarize active learning works that are specifically tailored to medical image analysis. Additionally, we conduct a thorough comparative analysis of the performance of different AL methods in medical image analysis with experiments. In the end, we offer our perspectives on the future trends and challenges of active learning and its applications in medical image analysis. An accompanying paper list and code for the comparative analysis is available in https://github.com/LightersWang/Awesome-Active-Learning-for-Medical-Image-Analysis.

Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis

The full acceptance of Deep Learning (DL) models in the clinical field is rather low with respect to the quantity of high-performing solutions reported in the literature. End users are particularly reluctant to rely on the opaque predictions of DL models. Uncertainty quantification methods have been proposed in the literature as a potential solution, to reduce the black-box effect of DL models and increase the interpretability and the acceptability of the result by the final user. In this review, we propose an overview of the existing methods to quantify uncertainty associated with DL predictions. We focus on applications to medical image analysis, which present specific challenges due to the high dimensionality of images and their variable quality, as well as constraints associated with real-world clinical routine. Moreover, we discuss the concept of structural uncertainty, a corpus of methods to facilitate the alignment of segmentation uncertainty estimates with clinical attention. We then discuss the evaluation protocols to validate the relevance of uncertainty estimates. Finally, we highlight the open challenges for uncertainty quantification in the medical field.

Automatic hip osteoarthritis grading with uncertainty estimation from computed tomography using digitally-reconstructed radiographs

PURPOSE

Progression of hip osteoarthritis (hip OA) leads to pain and disability, likely leading to surgical treatment such as hip arthroplasty at the terminal stage. The severity of hip OA is often classified using the Crowe and Kellgren-Lawrence (KL) classifications. However, as the classification is subjective, we aimed to develop an automated approach to classify the disease severity based on the two grades using digitally-reconstructed radiographs from CT images.

METHODS

Automatic grading of the hip OA severity was performed using deep learning-based models. The models were trained to predict the disease grade using two grading schemes, i.e., predicting the Crowe and KL grades separately, and predicting a new ordinal label combining both grades and representing the disease progression of hip OA. The models were trained in classification and regression settings. In addition, the model uncertainty was estimated and validated as a predictor of classification accuracy. The models were trained and validated on a database of 197 hip OA patients, and externally validated on 52 patients. The model accuracy was evaluated using exact class accuracy (ECA), one-neighbor class accuracy (ONCA), and balanced accuracy.

RESULTS

The deep learning models produced a comparable accuracy of approximately 0.65 (ECA) and 0.95 (ONCA) in the classification and regression settings. The model uncertainty was significantly larger in cases with large classification errors ( P < 6 e - 3 ).

CONCLUSIONS

In this study, an automatic approach for grading hip OA severity from CT images was developed. The models have shown comparable performance with high ONCA, which facilitates automated grading in large-scale CT databases and indicates the potential for further disease progression analysis. Classification accuracy was correlated with the model uncertainty, which would allow for the prediction of classification errors. The code will be made publicly available at https://github.com/NAIST-ICB/HipOA-Grading .

Cardiovascular Imaging-Derived Skeletal Muscle Mass Correlates With Fitness and Survival in Patients With Univentricular Circulation

Aims This study aims to retrospectively quantify skeletal muscle mass from cardiovascular imaging studies in total cavopulmonary connection (TCPC) patients and to correlate calculated muscle mass with clinical outcomes. Materials and methods Ninety-one TCPC patients at a mean age of 24.0 ±5.5 years (37 women; 40.7%) who underwent chest computed tomography (CT) or cardiac magnetic resonance imaging (MRI) as part of their follow-up were identified in a single-center database. The cross-sectional skeletal muscle index (SMI) at the Th4 and Th12 levels was calculated from CT images, and the dorsal skeletal muscle area (SMA) at the Th12 level was measured from an MRI. Results Calculated SMI at Th12 level was 38.0 (34.5; 42.0) cm2.m-2 or 89.6 (81.9; 101.6) % of predicted values. The median follow-up from CT was 5.9 (3.1; 8.5) years, and the composite endpoint (death N=5, heart transplant N=6) was reached in a total of 11 (26.8%) patients. Patients with SMI (Th12) less than 90% of predicted values had a hazard ratio of 5.8 (95% CI: 1.2; 28.3) (p=0.03) for endpoint achievement. In the MRI group, dorsal SMA at the Th12 level was 27.6 ±5.1 cm2 in men and 20.0 ±5.8 cm2 in women. Correlations were found between SMA/kg and peak oxygen uptake (VO2 peak) (r=0.48, p=0.0005) and fat-free mass (r=0.63, p<0.0001), respectively. Conclusions A low SMI at the Th12 level was associated with a higher risk of death or cardiac transplantation. Evaluation of skeletal muscle mass using cardiovascular imaging methods allows rapid identification of individuals at risk of sarcopenia.

Uncertainty-based Active Learning by Bayesian U-Net for Multi-label Cone-beam CT Segmentation

INTRODUCTION

Training of Artificial Intelligence (AI) for biomedical image analysis depends on large annotated datasets. This study assessed the efficacy of Active Learning (AL) strategies training AI models for accurate multilabel segmentation and detection of periapical lesions in cone-beam CTs (CBCTs) using a limited dataset.

METHODS

Limited field-of-view CBCT volumes (n = 20) were segmented by clinicians (clinician segmentation [CS]) and Bayesian U-Net-based AL strategies. Two AL functions, Bayesian Active Learning by Disagreement [BALD] and Max_Entropy [ME], were used for multilabel segmentation ("Lesion"-"Tooth Structure"-"Bone"-"Restorative Materials"-"Background"), and compared to a non-AL benchmark Bayesian U-Net function. The training-to-testing set ratio was 4:1. Comparisons between the AL and Bayesian U-Net functions versus CS were made by evaluating the segmentation accuracy with the Dice indices and lesion detection accuracy. The Kruskal-Wallis test was used to assess statistically significant differences.

RESULTS

The final training set contained 26 images. After 8 AL iterations, lesion detection sensitivity was 84.0% for BALD, 76.0% for ME, and 32.0% for Bayesian U-Net, which was significantly different (P < .0001; H = 16.989). The mean Dice index for all labels was 0.680 ± 0.155 for Bayesian U-Net and 0.703 ± 0.166 for ME after eight AL iterations, compared to 0.601 ± 0.267 for Bayesian U-Net over the mean of all iterations. The Dice index for "Lesion" was 0.504 for BALD and 0.501 for ME after 8 AL iterations, and at a maximum 0.288 for Bayesian U-Net.

CONCLUSIONS

Both AL strategies based on uncertainty quantification from Bayesian U-Net BALD, and ME, provided improved segmentation and lesion detection accuracy for CBCTs. AL may contribute to reducing extensive labeling needs for training AI algorithms for biomedical image analysis in dentistry.

Hybrid representation-enhanced sampling for Bayesian active learning in musculoskeletal segmentation of lower extremities

PURPOSE

Manual annotations for training deep learning models in auto-segmentation are time-intensive. This study introduces a hybrid representation-enhanced sampling strategy that integrates both density and diversity criteria within an uncertainty-based Bayesian active learning (BAL) framework to reduce annotation efforts by selecting the most informative training samples.

METHODS

The experiments are performed on two lower extremity datasets of MRI and CT images, focusing on the segmentation of the femur, pelvis, sacrum, quadriceps femoris, hamstrings, adductors, sartorius, and iliopsoas, utilizing a U-net-based BAL framework. Our method selects uncertain samples with high density and diversity for manual revision, optimizing for maximal similarity to unlabeled instances and minimal similarity to existing training data. We assess the accuracy and efficiency using dice and a proposed metric called reduced annotation cost (RAC), respectively. We further evaluate the impact of various acquisition rules on BAL performance and design an ablation study for effectiveness estimation.

RESULTS

In MRI and CT datasets, our method was superior or comparable to existing ones, achieving a 0.8% dice and 1.0% RAC increase in CT (statistically significant), and a 0.8% dice and 1.1% RAC increase in MRI (not statistically significant) in volume-wise acquisition. Our ablation study indicates that combining density and diversity criteria enhances the efficiency of BAL in musculoskeletal segmentation compared to using either criterion alone.

CONCLUSION

Our sampling method is proven efficient in reducing annotation costs in image segmentation tasks. The combination of the proposed method and our BAL framework provides a semi-automatic way for efficient annotation of medical image datasets.

Predictions of thumb, hand, and arm muscle parameters derived using force measurements of varying complexity and neural networks

Subject-specific musculoskeletal models are a promising avenue for personalized healthcare. However, current methods for producing personalized models require dense, biomechanical datasets that include expensive and time-consuming physiological measurements. For personalized models to be clinically useful, we must be able to rapidly generate models from simple, easy to collect data. In this context, the objective of this paper is to evaluate if and how simple data, namely height/weight and pinch force data, can be used to achieve model personalization via machine learning. Using simulated lateral pinch force measurements from a synthetic population of 40,000 randomly generated subjects, we train neural networks to estimate four Hill-type muscle model parameters and bone density. We compare parameter estimates to the true parameters of 10,000 additional synthetic subjects. We also generate new personalized models using the parameter estimates and perform new lateral pinch simulations to compare predicted forces using these personalized models to those generated using a baseline model. We demonstrate that increasing force measurement complexity reduces the root-mean-square error in the majority of parameter estimates. Additionally, musculoskeletal models using neural network-based parameter estimates provide up to an 80% reduction in absolute error in simulated forces when compared to a generic model. Thus, easily obtained force measurements may be suitable for personalizing models of the thumb, although extending the method to more tasks and models involving other joints likely requires additional measurements.

Bi-Graph Reasoning for Masticatory Muscle Segmentation From Cone-Beam Computed Tomography

Automated segmentation of masticatory muscles is a challenging task considering ambiguous soft tissue attachments and image artifacts of low-radiation cone-beam computed tomography (CBCT) images. In this paper, we propose a bi-graph reasoning model (BGR) for the simultaneous detection and segmentation of multi-category masticatory muscles from CBCTs. The BGR exploits the local and long-range interdependencies of regions of interest and category-specific prior knowledge of masticatory muscles by reasoning on the category graph and the region graph. The category graph of the learnable muscle prior knowledge handles high-level dependencies of muscle categories, enhancing the feature representation with noise-agnostic category knowledge. The region graph models both local and global dependencies of the candidate muscle regions of interest. The proposed BGR accommodates the high-level dependencies and enhances the region features in the presence of entangled soft tissue and image artifacts. We evaluated the proposed approach by segmenting masticatory muscles on clinically acquired CBCTs. Extensive experimental results show that the BGR effectively segments masticatory muscles with state-of-the-art accuracy.

A Deep Convolutional Gated Recurrent Unit for CT Image Reconstruction

Computed tomography (CT) is one of the most important medical imaging technologies in use today. Most commercial CT products use a technique known as the filtered backprojection (FBP) that is fast and can produce decent image quality when an X-ray dose is high. However, the FBP is not good enough on low-dose X-ray CT imaging because the CT image reconstruction problem becomes more stochastic. A more effective reconstruction technique proposed recently and implemented in a limited number of CT commercial products is an iterative reconstruction (IR). The IR technique is based on a Bayesian formulation of the CT image reconstruction problem with an explicit model of the CT scanning, including its stochastic nature, and a prior model that incorporates our knowledge about what a good CT image should look like. However, constructing such prior knowledge is more complicated than it seems. In this article, we propose a novel neural network for CT image reconstruction. The network is based on the IR formulation and constructed with a recurrent neural network (RNN). Specifically, we transform the gated recurrent unit (GRU) into a neural network performing CT image reconstruction. We call it "GRU reconstruction." This neural network conducts concurrent dual-domain learning. Many deep learning (DL)-based methods in medical imaging are single-domain learning, but dual-domain learning performs better because it learns from both the sinogram and the image domain. In addition, we propose backpropagation through stage (BPTS) as a new RNN backpropagation algorithm. It is similar to the backpropagation through time (BPTT) of an RNN; however, it is tailored for iterative optimization. Results from extensive experiments indicate that our proposed method outperforms conventional model-based methods, single-domain DL methods, and state-of-the-art DL techniques in terms of the root mean squared error (RMSE), the peak signal-to-noise ratio (PSNR), and the structure similarity (SSIM) and in terms of visual appearance.

Bone mineral density estimation from a plain X-ray image by learning decomposition into projections of bone-segmented computed tomography

Osteoporosis is a prevalent bone disease that causes fractures in fragile bones, leading to a decline in daily living activities. Dual-energy X-ray absorptiometry (DXA) and quantitative computed tomography (QCT) are highly accurate for diagnosing osteoporosis; however, these modalities require special equipment and scan protocols. To frequently monitor bone health, low-cost, low-dose, and ubiquitously available diagnostic methods are highly anticipated. In this study, we aim to perform bone mineral density (BMD) estimation from a plain X-ray image for opportunistic screening, which is potentially useful for early diagnosis. Existing methods have used multi-stage approaches consisting of extraction of the region of interest and simple regression to estimate BMD, which require a large amount of training data. Therefore, we propose an efficient method that learns decomposition into projections of bone-segmented QCT for BMD estimation under limited datasets. The proposed method achieved high accuracy in BMD estimation, where Pearson correlation coefficients of 0.880 and 0.920 were observed for DXA-measured BMD and QCT-measured BMD estimation tasks, respectively, and the root mean square of the coefficient of variation values were 3.27 to 3.79% for four measurements with different poses. Furthermore, we conducted extensive validation experiments, including multi-pose, uncalibrated-CT, and compression experiments toward actual application in routine clinical practice.

Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013-2023)

Uncertainty estimation in healthcare involves quantifying and understanding the inherent uncertainty or variability associated with medical predictions, diagnoses, and treatment outcomes. In this era of Artificial Intelligence (AI) models, uncertainty estimation becomes vital to ensure safe decision-making in the medical field. Therefore, this review focuses on the application of uncertainty techniques to machine and deep learning models in healthcare. A systematic literature review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our analysis revealed that Bayesian methods were the predominant technique for uncertainty quantification in machine learning models, with Fuzzy systems being the second most used approach. Regarding deep learning models, Bayesian methods emerged as the most prevalent approach, finding application in nearly all aspects of medical imaging. Most of the studies reported in this paper focused on medical images, highlighting the prevalent application of uncertainty quantification techniques using deep learning models compared to machine learning models. Interestingly, we observed a scarcity of studies applying uncertainty quantification to physiological signals. Thus, future research on uncertainty quantification should prioritize investigating the application of these techniques to physiological signals. Overall, our review highlights the significance of integrating uncertainty techniques in healthcare applications of machine learning and deep learning models. This can provide valuable insights and practical solutions to manage uncertainty in real-world medical data, ultimately improving the accuracy and reliability of medical diagnoses and treatment recommendations.

Automatic orbital segmentation using deep learning-based 2D U-net and accuracy evaluation: A retrospective study

The purpose of this study was to verify whether the accuracy of automatic segmentation (AS) of computed tomography (CT) images of fractured orbits using deep learning (DL) is sufficient for clinical application. In the surgery of orbital fractures, many methods have been reported to create a 3D anatomical model for use as a reference. However, because the orbit bone is thin and complex, creating a segmentation model for 3D printing is complicated and time-consuming. Here, the training of DL was performed using U-Net as the DL model, and the AS output was validated with Dice coefficients and average symmetry surface distance (ASSD). In addition, the AS output was 3D printed and evaluated for accuracy by four surgeons, each with over 15 years of clinical experience. One hundred twenty-five CT images were prepared, and manual orbital segmentation was performed in all cases. Ten orbital fracture cases were randomly selected as validation data, and the remaining 115 were set as training data. AS was successful in all cases, with good accuracy: Dice, 0.860 ± 0.033 (mean ± SD); ASSD, 0.713 ± 0.212 mm. In evaluating AS accuracy, the expert surgeons generally considered that it could be used for surgical support without further modification. The orbital AS algorithm developed using DL in this study is extremely accurate and can create 3D models rapidly at low cost, potentially enabling safer and more accurate surgeries.

Development and validation of an open-source tool for opportunistic screening of osteoporosis from hip CT images

Aims

This study aimed to develop and validate a fully automated system that quantifies proximal femoral bone mineral density (BMD) from CT images.

Methods

The study analyzed 978 pairs of hip CT and dual-energy X-ray absorptiometry (DXA) measurements of the proximal femur (DXA-BMD) collected from three institutions. From the CT images, the femur and a calibration phantom were automatically segmented using previously trained deep-learning models. The Hounsfield units of each voxel were converted into density (mg/cm3). Then, a deep-learning model trained by manual landmark selection of 315 cases was developed to select the landmarks at the proximal femur to rotate the CT volume to the neutral position. Finally, the CT volume of the femur was projected onto the coronal plane, and the areal BMD of the proximal femur (CT-aBMD) was quantified. CT-aBMD correlated to DXA-BMD, and a receiver operating characteristic (ROC) analysis quantified the accuracy in diagnosing osteoporosis.

Results

CT-aBMD was successfully measured in 976/978 hips (99.8%). A significant correlation was found between CT-aBMD and DXA-BMD (r = 0.941; p < 0.001). In the ROC analysis, the area under the curve to diagnose osteoporosis was 0.976. The diagnostic sensitivity and specificity were 88.9% and 96%, respectively, with the cutoff set at 0.625 g/cm2.

Conclusion

Accurate DXA-BMD measurements and diagnosis of osteoporosis were performed from CT images using the system developed herein. As the models are open-source, clinicians can use the proposed system to screen osteoporosis and determine the surgical strategy for hip surgery.

Deep learning-based pectoralis muscle volume segmentation method from chest computed tomography image using sagittal range detection and axial slice-based segmentation

The pectoralis muscle is an important indicator of respiratory muscle function and has been linked to various parenchymal biomarkers, such as airflow limitation severity and diffusing capacity for carbon monoxide, which are widely used in diagnosing parenchymal diseases, including asthma and chronic obstructive pulmonary disease. Pectoralis muscle segmentation is a method for measuring muscle volume and mass for various applications. The segmentation method is based on deep-learning techniques that combine a muscle area detection model and a segmentation model. The training dataset for the detection model comprised multichannel images of patients, whereas the segmentation model was trained on 7,796 cases of the computed tomography (CT) image dataset of 1,841 patients. The dataset was expanded incrementally through an active learning process. The performance of the model was evaluated by comparing the segmentation results with manual annotations by radiologists and the volumetric differences between the CT image datasets of the same patients. The results indicated that the machine learning model is promising in segmenting the pectoralis major muscle, with good agreement between the automatic segmentation and manual annotations by radiologists. The training accuracy and loss values of the validation set were 0.9954 and 0.0725, respectively, and for segmentation, the loss value was 0.0579. This study shows the potential clinical usefulness of the machine learning model for pectoralis major muscle segmentation as a quantitative biomarker for various parenchymal and muscular diseases.

A novel image augmentation based on statistical shape and intensity models: application to the segmentation of hip bones from CT images

BACKGROUND

The collection and annotation of medical images are hindered by data scarcity, privacy, and ethical reasons or limited resources, negatively affecting deep learning approaches. Data augmentation is often used to mitigate this problem, by generating synthetic images from training sets to improve the efficiency and generalization of deep learning models.

METHODS

We propose the novel use of statistical shape and intensity models (SSIM) to generate augmented images with variety in both shape and intensity of imaged structures and surroundings. The SSIM uses segmentations from training images to create co-registered tetrahedral meshes of the structures and to efficiently encode image intensity in their interior with Bernstein polynomials. In the context of segmentation of hip joint (pathological) bones from retrospective computed tomography images of 232 patients, we compared the impact of SSIM-based and basic augmentations on the performance of a U-Net model.

RESULTS

In a fivefold cross-validation, the SSIM augmentation improved segmentation robustness and accuracy. In particular, the combination of basic and SSIM augmentation outperformed trained models not using any augmentation, or relying exclusively on a simple form of augmentation, achieving Dice similarity coefficient and Hausdorff distance of 0.95 [0.93-0.96] and 6.16 [4.90-8.08] mm (median [25th-75th percentiles]), comparable to previous work on pathological hip segmentation.

CONCLUSIONS

We proposed a novel augmentation varying both the shape and appearance of structures in generated images. Tested on bone segmentation, our approach is generalizable to other structures or tasks such as classification, as long as SSIM can be built from training data.

RELEVANCE STATEMENT

Our data augmentation approach produces realistic shape and appearance variations of structures in generated images, which supports the clinical adoption of AI in radiology by alleviating the collection of clinical imaging data and by improving the performance of AI applications.

KEY POINTS

• Data augmentation generally improves the accuracy and generalization of deep learning models. • Traditional data augmentation does not consider the appearance of imaged structures. • Statistical shape and intensity models (SSIM) synthetically generate variations of imaged structures. • SSIM support novel augmentation approaches, demonstrated with computed tomography bone segmentation.

Assisted annotation in Deep LOGISMOS: Simultaneous multi-compartment 3D MRI segmentation of calf muscles

BACKGROUND

Automated segmentation of individual calf muscle compartments in 3D MR images is gaining importance in diagnosing muscle disease, monitoring its progression, and prediction of the disease course. Although deep convolutional neural networks have ushered in a revolution in medical image segmentation, achieving clinically acceptable results is a challenging task and the availability of sufficiently large annotated datasets still limits their applicability.

PURPOSE

In this paper, we present a novel approach combing deep learning and graph optimization in the paradigm of assisted annotation for solving general segmentation problems in 3D, 4D, and generally n-D with limited annotation cost.

METHODS

Deep LOGISMOS combines deep-learning-based pre-segmentation of objects of interest provided by our convolutional neural network, FilterNet+, and our 3D multi-objects LOGISMOS framework (layered optimal graph image segmentation of multiple objects and surfaces) that uses newly designed trainable machine-learned cost functions. In the paradigm of assisted annotation, multi-object JEI for efficient editing of automated Deep LOGISMOS segmentation was employed to form a new larger training set with significant decrease of manual tracing effort.

RESULTS

We have evaluated our method on 350 lower leg (left/right) T1-weighted MR images from 93 subjects (47 healthy, 46 patients with muscular morbidity) by fourfold cross-validation. Compared with the fully manual annotation approach, the annotation cost with assisted annotation is reduced by 95%, from 8 h to 25 min in this study. The experimental results showed average Dice similarity coefficient (DSC) of96.56 ± 0.26 % $96.56\pm 0.26 \%$ and average absolute surface positioning error of 0.63 pixels (0.44 mm) for the five 3D muscle compartments for each leg. These results significantly improve our previously reported method and outperform the state-of-the-art nnUNet method.

CONCLUSIONS

Our proposed approach can not only dramatically reduce the expert's annotation efforts but also significantly improve the segmentation performance compared to the state-of-the-art nnUNet method. The notable performance improvements suggest the clinical-use potential of our new fully automated simultaneous segmentation of calf muscle compartments.

Optimization of Revision Hip Arthroplasty Workflow by Means of Detailed Pre-Surgical Planning Using Computed Tomography Data, Open-Source Software and Three-Dimensional-Printed Models

BACKGROUND

In revision hip arthroplasty (RHA), establishing the center of rotation (COR) can be technically challenging due to the acetabular bone destruction that is usually present, particularly in severe cases such as Paprosky type II and III defects. The aim of this study was to demonstrate the use of open-source medical image reconstruction software and low-cost 3D anatomical models in pre-surgical planning of RHA.

METHODS

A total of 10 patients, underwent RHA and were included in the study. Computed tomography (CT) scans were performed for all cases, before surgery and approximately 1 week after the procedure. The reconstruction of CT data, 3D virtual planning of the COR and positioning of acetabular cups, including their inclination and anteversion angles, was carried out using the free open source software platform 3D Slicer. In addition, anatomical models of the pelvis were built on a desktop 3D printer from polylactic acid (PLA). Preoperative and postoperative reconstructed imaging data were compared for each patient, and the position of the acetabular cups as well as the COR were evaluated for each case.

RESULTS

Analysis of the pre- and post-op center of rotation position data indicated statistically insignificant differences for the location of the COR on the X-axis (1.5 mm, t = 0.5741, p = 0.5868) with a fairly strong correlation of the results (r = -0.672, p = 0.0982), whilst for the location of the COR in the Y and Z-axes, there was statistical dependence (Y axis, 4.7 mm, t = 3.168 and p = 0.0194; Z axis, 1.9 mm, t = 1.887 and p = 0.1081). A strong correlation for both axes was also observed (Y and Z) (Y-axis, r = 0.9438 and p = 0.0014; Z-axis, r = 0.8829 and p = 0.0084). Analysis of inclination angle values showed a statistically insignificant difference between mean values (3.9 degrees, t = 1.111, p = 0.3092) and a moderate correlation was found between mean values (r = -0.4042, p = 0.3685). Analysis of the anteversion angle showed a statistically insignificant difference between mean values (1.9 degrees, t = 0.8671, p = 0.4192), while a moderate correlation between mean values was found (r = -0.4782, p = 0.2777).

CONCLUSIONS

Three-dimensional reconstruction software, together with low-cost anatomical models, are very effective tools for pre-surgical planning, which have great potential use in orthopedic surgery, particularly RHA. In up and in- and up and out-type defects, it is essential to establish a new COR and to identify three support points within the revision acetabulum in order to correctly position acetabular cups.

Deep-learning-based automatic facial bone segmentation using a two-dimensional U-Net

The use of deep learning (DL) in medical imaging is becoming increasingly widespread. Although DL has been used previously for the segmentation of facial bones in computed tomography (CT) images, there are few reports of segmentation involving multiple areas. In this study, a U-Net was used to investigate the automatic segmentation of facial bones into eight areas, with the aim of facilitating virtual surgical planning (VSP) and computer-aided design and manufacturing (CAD/CAM) in maxillofacial surgery. CT data from 50 patients were prepared and used for training, and five-fold cross-validation was performed. The output results generated by the DL model were validated by Dice coefficient and average symmetric surface distance (ASSD). The automatic segmentation was successful in all cases, with a mean± standard deviation Dice coefficient of 0.897 ± 0.077 and ASSD of 1.168 ± 1.962 mm. The accuracy was very high for the mandible (Dice coefficient 0.984, ASSD 0.324 mm) and zygomatic bones (Dice coefficient 0.931, ASSD 0.487 mm), and these could be introduced for VSP and CAD/CAM without any modification. The results for other areas, particularly the teeth, were slightly inferior, with possible reasons being the effects of defects, bonded maxillary and mandibular teeth, and metal artefacts. A limitation of this study is that the data were from a single institution. Hence further research is required to improve the accuracy for some facial areas and to validate the results in larger and more diverse populations.

Near-Infrared Blood Vessel Image Segmentation Using Background Subtraction and Improved Mathematical Morphology

The precise display of blood vessel information for doctors is crucial. This is not only true for facilitating intravenous injections, but also for the diagnosis and analysis of diseases. Currently, infrared cameras can be used to capture images of superficial blood vessels. However, their imaging quality always has the problems of noises, breaks, and uneven vascular information. In order to overcome these problems, this paper proposes an image segmentation algorithm based on the background subtraction and improved mathematical morphology. The algorithm regards the image as a superposition of blood vessels into the background, removes the noise by calculating the size of connected domains, achieves uniform blood vessel width, and smooths edges that reflect the actual blood vessel state. The algorithm is evaluated subjectively and objectively in this paper to provide a basis for vascular image quality assessment. Extensive experimental results demonstrate that the proposed method can effectively extract accurate and clear vascular information.

Deep Learning-Based Medical Images Segmentation of Musculoskeletal Anatomical Structures: A Survey of Bottlenecks and Strategies

By leveraging the recent development of artificial intelligence algorithms, several medical sectors have benefited from using automatic segmentation tools from bioimaging to segment anatomical structures. Segmentation of the musculoskeletal system is key for studying alterations in anatomical tissue and supporting medical interventions. The clinical use of such tools requires an understanding of the proper method for interpreting data and evaluating their performance. The current systematic review aims to present the common bottlenecks for musculoskeletal structures analysis (e.g., small sample size, data inhomogeneity) and the related strategies utilized by different authors. A search was performed using the PUBMED database with the following keywords: deep learning, musculoskeletal system, segmentation. A total of 140 articles published up until February 2022 were obtained and analyzed according to the PRISMA framework in terms of anatomical structures, bioimaging techniques, pre/post-processing operations, training/validation/testing subset creation, network architecture, loss functions, performance indicators and so on. Several common trends emerged from this survey; however, the different methods need to be compared and discussed based on each specific case study (anatomical region, medical imaging acquisition setting, study population, etc.). These findings can be used to guide clinicians (as end users) to better understand the potential benefits and limitations of these tools.

Artificial intelligence-based volumetric analysis of muscle atrophy and fatty degeneration in patients with hip osteoarthritis and its correlation with health-related quality of life

PURPOSE

Artificial intelligence (AI) technologies have enabled precise three-dimensional analysis of individual muscles on computed tomography (CT) or magnetic resonance images via automatic segmentation. This study aimed to perform three-dimensional assessments of pelvic and thigh muscle atrophy and fatty degeneration in patients with unilateral hip osteoarthritis using CT and to evaluate the correlation with health-related quality of life (HRQoL).

METHODS

The study included one man and 43 women. Six muscle groups were segmented, and the muscle atrophy ratio was calculated volumetrically. The degree of fatty degeneration was defined as the difference between the mean CT values (Hounsfield units [HU]) of the healthy and affected sides. HRQoL was evaluated using the Western Ontario and McMaster Universities Osteoarthritis (WOMAC) index and the Japanese Orthopaedic Association Hip Disease Evaluation Questionnaire (JHEQ).

RESULTS

The mean muscle atrophy rate was 16.3%, and the mean degree of muscle fatty degeneration was 7.9 HU. Multivariate correlation analysis revealed that the WOMAC stiffness subscale was significantly related to fatty degeneration of the hamstrings, the WOMAC physical function subscale was significantly related to fatty degeneration of the iliopsoas muscle, and the JHEQ movement subscale was significantly related to fatty degeneration of the hip adductors.

CONCLUSION

We found that fatty degeneration of the hamstrings, iliopsoas, and hip adductor muscles was significantly related to HRQoL in patients with hip osteoarthritis. These findings suggest that these muscles should be targeted during conservative rehabilitation for HOA and perioperative rehabilitation for THA.

Automated 3-dimensional MRI segmentation for the posterosuperior rotator cuff tear lesion using deep learning algorithm

INTRODUCTION

Rotator cuff tear (RCT) is a challenging and common musculoskeletal disease. Magnetic resonance imaging (MRI) is a commonly used diagnostic modality for RCT, but the interpretation of the results is tedious and has some reliability issues. In this study, we aimed to evaluate the accuracy and efficacy of the 3-dimensional (3D) MRI segmentation for RCT using a deep learning algorithm.

METHODS

A 3D U-Net convolutional neural network (CNN) was developed to detect, segment, and visualize RCT lesions in 3D, using MRI data from 303 patients with RCTs. The RCT lesions were labeled by two shoulder specialists in the entire MR image using in-house developed software. The MRI-based 3D U-Net CNN was trained after the augmentation of a training dataset and tested using randomly selected test data (training: validation: test data ratio was 6:2:2). The segmented RCT lesion was visualized in a three-dimensional reconstructed image, and the performance of the 3D U-Net CNN was evaluated using the Dice coefficient, sensitivity, specificity, precision, F1-score, and Youden index.

RESULTS

A deep learning algorithm using a 3D U-Net CNN successfully detected, segmented, and visualized the area of RCT in 3D. The model's performance reached a 94.3% of Dice coefficient score, 97.1% of sensitivity, 95.0% of specificity, 84.9% of precision, 90.5% of F1-score, and Youden index of 91.8%.

CONCLUSION

The proposed model for 3D segmentation of RCT lesions using MRI data showed overall high accuracy and successful 3D visualization. Further studies are necessary to determine the feasibility of its clinical application and whether its use could improve care and outcomes.

Deep learning-based fully automated body composition analysis of thigh CT: comparison with DXA measurement

OBJECTIVES

To compare volumetric CT with DL-based fully automated segmentation and dual-energy X-ray absorptiometry (DXA) in the measurement of thigh tissue composition.

METHODS

This prospective study was performed from January 2019 to December 2020. The participants underwent DXA to determine the body composition of the whole body and thigh. CT was performed in the thigh region; the images were automatically segmented into three muscle groups and adipose tissue by custom-developed DL-based automated segmentation software. Subsequently, the program reported the tissue composition of the thigh. The correlation and agreement between variables measured by DXA and CT were assessed. Then, CT thigh tissue volume prediction equations based on DXA-derived thigh tissue mass were developed using a general linear model.

RESULTS

In total, 100 patients (mean age, 44.9 years; 60 women) were evaluated. There was a strong correlation between the CT and DXA measurements (R = 0.813~0.98, p < 0.001). There was no significant difference in total soft tissue mass between DXA and CT measurement (p = 0.183). However, DXA overestimated thigh lean (muscle) mass and underestimated thigh total fat mass (p < 0.001). The DXA-derived lean mass was an average of 10% higher than the CT-derived lean mass and 47% higher than the CT-derived lean muscle mass. The DXA-derived total fat mass was approximately 20% lower than the CT-derived total fat mass. The predicted CT tissue volume using DXA-derived data was highly correlated with actual CT-measured tissue volume in the validation group (R² = 0.96~0.97, p < 0.001).

CONCLUSIONS

Volumetric CT measurements with DL-based fully automated segmentation are a rapid and more accurate method for measuring thigh tissue composition.

KEY POINTS

• There was a positive correlation between CT and DXA measurements in both the whole body and thigh. • DXA overestimated thigh lean mass by 10%, lean muscle mass by 47%, but underestimated total fat mass by 20% compared to the CT method. • The equations for predicting CT volume (cm³) were developed using DXA data (g), age, height (cm), and body weight (kg) and good model performance was proven in the validation study.

The Effect of Region of Interest on Measurement of Bone Mineral Density of the Proximal Femur: Simulation Analysis Using CT Images

While accurate measurement of bone mineral density (BMD) is essential in the diagnosis of osteoporosis and in evaluating the treatment of osteoporosis, it is unclear how region of interest (ROI) settings affect measurement of BMD at the total proximal femur region. In this study, we performed a simulation analysis to clarify the effect on BMD measurement of changing the ROI using hip computed tomography (CT) images of 75 females (mean age, 62.4 years). Digitally reconstructed radiographs of the proximal femur region were generated from CT images to calculate the change in BMD when the proximal boundary of the ROI was altered by 0-10 mm, and when the distal boundary of the ROI was altered by 0-30 mm. Further, changes in BMD were compared across BMD classification groups. A mean BMD increase of 0.62% was found for each 1-mm extension of the distal boundary. A mean BMD decrease of 0.18% was found for each 1-mm alteration of the proximal boundary. Comparing BMD classification groups, patients with osteoporosis and osteopenia demonstrated greater BMD changes than patients with normal BMD for the distal boundary (0.68%, 0.64%, and 0.54%, respectively) and patients with osteoporosis demonstrated greater BMD changes than patients with osteoporosis and normal BMD for the proximal boundary (0.37%, 0.13%, and 0.03%, respectively). In conclusion, our study found that a consistent ROI setting, especially on the distal boundary, is necessary for the accurate measurement of total proximal femur BMD. Based on the findings, we recommend confirming that the ROI setting shown on the BMD result form is consistent with changes in serial BMD.

Ensemble learning via supervision augmentation for white matter hyperintensity segmentation

Since the ambiguous boundary of the lesion and inter-observer variability, white matter hyperintensity segmentation annotations are inherently noisy and uncertain. On the other hand, the high capacity of deep neural networks (DNN) enables them to overfit labels with noise and uncertainty, which may lead to biased models with weak generalization ability. This challenge has been addressed by leveraging multiple annotations per image. However, multiple annotations are often not available in a real-world scenario. To mitigate the issue, this paper proposes a supervision augmentation method (SA) and combines it with ensemble learning (SA-EN) to improve the generalization ability of the model. SA can obtain diverse supervision information by estimating the uncertainty of annotation in a real-world scenario that per image have only one ambiguous annotation. Then different base learners in EN are trained with diverse supervision information. The experimental results on two white matter hyperintensity segmentation datasets demonstrate that SA-EN gets the optimal accuracy compared with other state-of-the-art ensemble methods. SA-EN is more effective on small datasets, which is more suitable for medical image segmentation with few annotations. A quantitative study is presented to show the effect of ensemble size and the effectiveness of the ensemble model. Furthermore, SA-EN can capture two types of uncertainty, aleatoric uncertainty modeled in SA and epistemic uncertainty modeled in EN.

Automatic Segmentation of Magnetic Resonance Images of Severe Patients with Advanced Liver Cancer and the Molecular Mechanism of Emodin-Induced Apoptosis of HepG2 Cells under the Deep Learning

To improve the accuracy of clinical diagnosis of severe patients with advanced liver cancer and enhance the effect of chemotherapy treatment, the U-Net model was optimized by introducing the batch normalization (BN) layer and the dropout layer, and the segmentation training and verification of the optimized model were realized by the magnetic resonance (MR) image data. Subsequently, HepG2 cells were taken as the research objects and treated with 0, 10, 20, 40, 60, 80, and 100 μmol/L emodin (EMO), respectively. The methyl thiazolyl tetrazolium (MTT) method was used to explore the changes in cell viability, the acridine orange (AO)/ethidium bromide (EB) and 4',6-diamidino-2-phenylindole (DAPI) were used for staining, the Annexin V fluorescein isothiocyanate (FITC)/propidium iodide (PI) (Annexin V-FITC/PI) was adopted to detect the apoptosis after EMO treatment, and the Western blot (WB) method was used with the purpose of exploring the changes in protein expression levels of PARP, Bcl-2, and p53 in the cells after treatment. It was found that compared with the original U-Net model, the introduction of the BN layer and the dropout layer can improve the robustness of the U-Net model, and the optimized U-Net model had the highest dice similarity coefficient (DSC) (98.45%) and mean average precision (MAP) (0.88) for the liver tumor segmentation.

Utility of a novel integrated deep convolutional neural network for the segmentation of hip joint from computed tomography images in the preoperative planning of total hip arthroplasty

Purpose

Preoperative three-dimensional planning is important for total hip arthroplasty. To simulate the placement of joint implants on computed tomography (CT), pelvis and femur must be segmented. Accurate and rapid segmentation of the hip joint is challenging. This study aimed to develop a novel deep learning network, named Changmugu Net (CMG Net), which could achieve accurate segmentation of the femur and pelvis.

Methods

The overall deep neural network architecture of CMG Net employed three interrelated modules. CMG Net included the 2D U-net to separate the bony and soft tissues. The modular hierarchy method was used for the main femur segmentation to achieve better performance. A layer classifier was adopted to localise femur layers among a series of CT scan images. The first module was a modified 2D U-net, which separated bony and soft tissues; it provided intermediate supervision for the main femur segmentation. The second module was the main femur segmentation, which was used to distinguish the femur from the acetabulum. The third module was the layer classifier, which served as a post-processor for the second module.

Results

There was a much greater overlap in accuracy results with the “gold standard” segmentation than with competing networks. The dice overlap coefficient was 93.55% ± 5.57%; the mean surface distance was 1.34 ± 0.24 mm, and the Hausdorff distance was 4.19 ± 1.04 mm in the normal and diseased hips, which indicated greater accuracy than the other four competing networks. Moreover, the mean segmentation time of CMG Net was 25.87 ± 2.73 s, which was shorter than the times of the other four networks.

Conclusions

The prominent segmentation accuracy and run-time of CMG Net suggest that it is a reliable method for clinicians to observe anatomical structures of the hip joints, even in severely diseased cases.

Modality specific U-Net variants for biomedical image segmentation: a survey

With the advent of advancements in deep learning approaches, such as deep convolution neural network, residual neural network, adversarial network; U-Net architectures are most widely utilized in biomedical image segmentation to address the automation in identification and detection of the target regions or sub-regions. In recent studies, U-Net based approaches have illustrated state-of-the-art performance in different applications for the development of computer-aided diagnosis systems for early diagnosis and treatment of diseases such as brain tumor, lung cancer, alzheimer, breast cancer, etc., using various modalities. This article contributes in presenting the success of these approaches by describing the U-Net framework, followed by the comprehensive analysis of the U-Net variants by performing (1) inter-modality, and (2) intra-modality categorization to establish better insights into the associated challenges and solutions. Besides, this article also highlights the contribution of U-Net based frameworks in the ongoing pandemic, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) also known as COVID-19. Finally, the strengths and similarities of these U-Net variants are analysed along with the challenges involved in biomedical image segmentation to uncover promising future research directions in this area.

An Uncertainty-based Neural Network for Explainable Trajectory Segmentation

As a variant task of time-series segmentation, trajectory segmentation is a key task in the applications of transportation pattern recognition and traffic analysis. However, segmenting trajectory is faced with challenges of implicit patterns and sparse results. Although deep neural networks have tremendous advantages in terms of high-level feature learning performance, deploying as a blackbox seriously limits the real-world applications. Providing explainable segmentations has significance for result evaluation and decision making. Thus, in this article, we address trajectory segmentation by proposing a Bayesian Encoder-Decoder Network (BED-Net) to provide accurate detection with explainability and references for the following active-learning procedures. BED-Net consists of a segmentation module based on Monte Carlo dropout and an explanation module based on uncertainty learning that provides results evaluation and visualization. Experimental results on both benchmark and real-world datasets indicate that BED-Net outperforms the rival methods and offers excellent explainability in the applications of trajectory segmentation.

Probabilistic spatial analysis in quantitative microscopy with uncertainty-aware cell detection using deep Bayesian regression

The investigation of biological systems with three-dimensional microscopy demands automatic cell identification methods that not only are accurate but also can imply the uncertainty in their predictions. The use of deep learning to regress density maps is a popular successful approach for extracting cell coordinates from local peaks in a postprocessing step, which then, however, hinders any meaningful probabilistic output. We propose a framework that can operate on large microscopy images and output probabilistic predictions (i) by integrating deep Bayesian learning for the regression of uncertainty-aware density maps, where peak detection algorithms generate cell proposals, and (ii) by learning a mapping from prediction proposals to a probabilistic space that accurately represents the chances of a successful prediction. Using these calibrated predictions, we propose a probabilistic spatial analysis with Monte Carlo sampling. We demonstrate this in a bone marrow dataset, where our proposed methods reveal spatial patterns that are otherwise undetectable.

Development of an open-source measurement system to assess the areal bone mineral density of the proximal femur from clinical CT images

Commercial software is generally needed to measure the areal bone mineral density (aBMD) of the proximal femur from clinical computed tomography (CT) images. This study developed and verified an open-source reproducible system to quantify CT-aBMD to screen osteoporosis using clinical CT images.

PURPOSE

For existing CT images acquired for various reasons other than osteoporosis, it might be beneficial to estimate areal BMD as assessed by dual-energy X-ray absorptiometry (DXA-based BMD) to ascertain the bone status based on DXA. In this study, we aimed to (1) develop an open-source reproducible measurement system to quantify DXA-based BMD from CT images and (2) validate its accuracy.

METHODS

This study analyzed 75 pairs of hip CT and DXA images of women that were acquired for the preoperative assessment of total hip arthroplasty. From the CT images, the femur and a calibration phantom were automatically segmented using pre-trained codes/models available at https://github.com/keisuke-uemura . The proximal femoral region was isolated by manually selected landmarks and was projected onto the coronal plane to measure the areal density (CT-aHU). The calibration phantom was employed to convert the CT-aHU into CT-aBMD. Each parameter was correlated with DXA-based BMD, and the residual errors of CT images to estimate the T-scores in DXA were calculated using the standard error of estimate (SEE).

RESULTS

The correlation coefficients of DXA-based BMD with CT-aHU and CT-aBMD were 0.947 and 0.950, respectively (both p < 0.001). The SEE for quantifying the T-scores in DXA were 0.51 and 0.50 for CT-aHU and CT-aBMD, respectively.

CONCLUSION

With the method developed herein, CT permits estimation of the DXA-based BMD of the proximal femur within the standard DXA total hip region of interest with an SEE of 0.5 in T-scores. The radiation dose for CT acquisition needs consideration; therefore, our data do not provide a rationale for performing CT for screening osteoporosis. However, on CT images already acquired for clinical indications other than osteoporosis, researchers may use this open-source system to investigate osteoporosis status through the estimated DXA-based BMD of the proximal femur.

Deep Learning for Orthopedic Disease Based on Medical Image Analysis: Present and Future

Since its development, deep learning has been quickly incorporated into the field of medicine and has had a profound impact. Since 2017, many studies applying deep learning-based diagnostics in the field of orthopedics have demonstrated outstanding performance. However, most published papers have focused on disease detection or classification, leaving some unsatisfactory reports in areas such as segmentation and prediction. This review introduces research published in the field of orthopedics classified according to disease from the perspective of orthopedic surgeons, and areas of future research are discussed. This paper provides orthopedic surgeons with an overall understanding of artificial intelligence-based image analysis and the information that medical data should be treated with low prejudice, providing developers and researchers with insight into the real-world context in which clinicians are embracing medical artificial intelligence.

Automatic quadriceps and patellae segmentation of MRI with cascaded U2 -Net and SASSNet deep learning model

PURPOSE

Automatic muscle segmentation is critical for advancing our understanding of human physiology, biomechanics, and musculoskeletal pathologies, as it allows for timely exploration of large multi-dimensional image sets. Segmentation models are rarely developed/validated for the pediatric model. As such, autosegmentation is not available to explore how muscle architectural changes during development and how disease/pathology affects the developing musculoskeletal system. Thus, we aimed to develop and validate an end-to-end, fully automated, deep learning model for accurate segmentation of the rectus femoris and vastus lateral, medialis, and intermedialis using a pediatric database.

METHODS

We developed a two-stage cascaded deep learning model in a coarse-to-fine manner. In the first stage, the U² -Net roughly detects the muscle subcompartment region. Then, in the second stage, the shape-aware 3D semantic segmentation method SASSNet refines the cropped target regions to generate the more finer and accurate segmentation masks. We utilized multifeature image maps in both stages to stabilize performance and validated their use with an ablation study. The second-stage SASSNet was independently run and evaluated with three different cropped region resolutions: the original image resolution, and images downsampled 2× and 4× (high, mid, and low). The relationship between image resolution and segmentation accuracy was explored. In addition, the patella was included as a comparator to past work. We evaluated segmentation accuracy using leave-one-out testing on a database of 3D MR images (0.43 × 0.43 × 2 mm) from 40 pediatric participants (age 15.3 ± 1.9 years, 55.8 ± 11.8 kg, 164.2 ± 7.9 cm, 38F/2 M).

RESULTS

The mid-resolution second stage produced the best results for the vastus medialis, rectus femoris, and patella (Dice similarity coefficient = 95.0%, 95.1%, 93.7%), whereas the low-resolution second stage produced the best results for the vastus lateralis and vastus intermedialis (DSC = 94.5% and 93.7%). In comparing the low- to mid-resolution cases, the vasti intermedialis, vastus medialis, rectus femoris, and patella produced significant differences (p = 0.0015, p = 0.0101, p < 0.0001, p = 0.0003) and the vasti lateralis did not (p = 0.2177). The high-resolution stage 2 had significantly lower accuracy (1.0 to 4.4 dice percentage points) compared to both the mid- and low-resolution routines (p value ranged from < 0.001 to 0.04). The one exception was the rectus femoris, where there was no difference between the low- and high-resolution cases. The ablation study demonstrated that the multifeature is more reliable than the single feature.

CONCLUSIONS

Our successful implementation of this two-stage segmentation pipeline provides a critical tool for expanding pediatric muscle physiology and clinical research. With a relatively small and variable dataset, our fully automatic segmentation technique produces accuracies that matched or exceeded the current state of the art. The two-stage segmentation avoids memory issues and excessive run times by using a first stage focused on cropping out unnecessary data. The excellent Dice similarity coefficients improve upon previous template-based automatic and semiautomatic methodologies targeting the leg musculature. More importantly, with a naturally variable dataset (size, shape, etc.), the proposed model demonstrates slightly improved accuracies, compared to previous neural networks methods.

Deep learning methods for automatic segmentation of lower leg muscles and bones from MRI scans of children with and without cerebral palsy

Cerebral palsy is a neurological condition that is known to affect muscle growth. Detailed investigations of muscle growth require segmentation of muscles from MRI scans, which is typically done manually. In this study, we evaluated the performance of 2D, 3D, and hybrid deep learning models for automatic segmentation of 11 lower leg muscles and two bones from MRI scans of children with and without cerebral palsy. All six models were trained and evaluated on manually segmented T₁ -weighted MRI scans of the lower legs of 20 children, six of whom had cerebral palsy. The segmentation results were assessed using the median Dice similarity coefficient (DSC), average symmetric surface distance (ASSD), and volume error (VError) of all 13 labels of every scan. The best performance was achieved by H-DenseUNet, a hybrid model (DSC 0.90, ASSD 0.5 mm, and VError 2.6 cm³ ). The performance was equivalent to the inter-rater performance of manual segmentation (DSC 0.89, ASSD 0.6 mm, and VError 3.3 cm³ ). Models trained with the Dice loss function outperformed models trained with the cross-entropy loss function. Near-optimal performance could be attained using only 11 scans for training. Segmentation performance was similar for scans of typically developing children (DSC 0.90, ASSD 0.5 mm, and VError 2.8 cm³ ) and children with cerebral palsy (DSC 0.85, ASSD 0.6 mm, and VError 2.4 cm³ ). These findings demonstrate the feasibility of fully automatic segmentation of individual muscles and bones from MRI scans of children with and without cerebral palsy.

Deep generative models for automated muscle segmentation in computed tomography scanning

Accurate gluteus medius (GMd) volume evaluation may aid in the analysis of muscular atrophy states and help gain an improved understanding of patient recovery via rehabilitation. However, the segmentation of muscle regions in GMd images for cubic muscle volume assessment is time-consuming and labor-intensive. This study automated GMd-region segmentation from the computed tomography (CT) images of patients diagnosed with hip osteoarthritis using deep learning and evaluated the segmentation accuracy. To this end, 5250 augmented pairs of training data were obtained from five participants, and a conditional generative adversarial network was used to identify the relationships between the image pairs. Using the preserved test datasets, the results of automatic segmentation with the trained deep learning model were compared to those of manual segmentation in terms of the dice similarity coefficient (DSC), volume similarity (VS), and shape similarity (MS). As observed, the average DSC values for automatic and manual segmentations were 0.748 and 0.812, respectively, with a significant difference (p < 0.0001); the average VS values were 0.247 and 0.203, respectively, with no significant difference (p = 0.069); and the average MS values were 1.394 and 1.156, respectively, with no significant difference (p = 0.308). The GMd volumes obtained by automatic and manual segmentation were 246.2 cm³ and 282.9 cm³, respectively. The noninferiority of the DSC obtained by automatic segmentation was verified against that obtained by manual segmentation. Accordingly, the proposed GAN-based automatic GMd-segmentation technique is confirmed to be noninferior to manual segmentation. Therefore, the findings of this research confirm that the proposed method not only reduces time and effort but also facilitates accurate assessment of the cubic muscle volume.

Translation of Cellular Protein Localization Using Convolutional Networks

Protein localization in cells has been analyzed by fluorescent labeling using indirect immunofluorescence and fluorescent protein tagging. However, the relationships between the localization of different proteins had not been analyzed using artificial intelligence. Here, we applied convolutional networks for the prediction of localization of the cytoskeletal proteins from the localization of the other proteins. Lamellipodia are one of the actin-dependent subcellular structures involved in cell migration and are mainly generated by the Wiskott-Aldrich syndrome protein (WASP)-family verprolin homologous protein 2 (WAVE2) and the membrane remodeling I-BAR domain protein IRSp53. Focal adhesion is another actin-based structure that contains vinculin protein and promotes lamellipodia formation and cell migration. In contrast, microtubules are not directly related to actin filaments. The convolutional network was trained using images of actin filaments paired with WAVE2, IRSp53, vinculin, and microtubules. The generated images of WAVE2, IRSp53, and vinculin were highly similar to their real images. In contrast, the microtubule images generated from actin filament images were inferior without the generation of filamentous structures, suggesting that microscopic images of actin filaments provide more information about actin-related protein localization. Collectively, this study suggests that image translation by the convolutional network can predict the localization of functionally related proteins, and the convolutional network might be used to describe the relationships between the proteins by their localization.

Scale- and Slice-aware Net (S2 aNet) for 3D segmentation of organs and musculoskeletal structures in pelvic MRI

PURPOSE

MRI of organs and musculoskeletal structures in the female pelvis presents a unique display of pelvic anatomy. Automated segmentation of pelvic structures plays an important role in personalized diagnosis and treatment on pelvic structures disease. Pelvic organ systems are very complicated, and it is a challenging task for 3D segmentation of massive pelvic structures on MRI.

METHODS

A new Scale- and Slice-aware Net ( S 2 aNet) is presented for 3D dense segmentation of 54 organs and musculoskeletal structures in female pelvic MR images. A Scale-aware module is designed to capture the spatial and semantic information of different-scale structures. A Slice-aware module is introduced to model similar spatial relationships of consecutive slices in 3D data. Moreover, S 2 aNet leverages a weight-adaptive loss optimization strategy to reinforce the supervision with more discriminative capability on hard samples and categories.

RESULTS

Experiments have been performed on a pelvic MRI cohort of 27 MR images from 27 patient cases. Across the cohort and 54 categories of organs and musculoskeletal structures manually delineated, S 2 aNet was shown to outperform the UNet framework and other state-of-the-art fully convolutional networks in terms of sensitivity, Dice similarity coefficient and relative volume difference.

CONCLUSION

The experimental results on the pelvic 3D MR dataset show that the proposed S 2 aNet achieves excellent segmentation results compared to other state-of-the-art models. To our knowledge, S 2 aNet is the first model to achieve 3D dense segmentation for 54 musculoskeletal structures on pelvic MRI, which will be leveraged to the clinical application under the support of more cases in the future.

Automated segmentation of an intensity calibration phantom in clinical CT images using a convolutional neural network

PURPOSE

In quantitative computed tomography (CT), manual selection of the intensity calibration phantom's region of interest is necessary for calculating density (mg/cm³) from the radiodensity values (Hounsfield units: HU). However, as this manual process requires effort and time, the purposes of this study were to develop a system that applies a convolutional neural network (CNN) to automatically segment intensity calibration phantom regions in CT images and to test the system in a large cohort to evaluate its robustness.

METHODS

This cross-sectional, retrospective study included 1040 cases (520 each from two institutions) in which an intensity calibration phantom (B-MAS200, Kyoto Kagaku, Kyoto, Japan) was used. A training dataset was created by manually segmenting the phantom regions for 40 cases (20 cases for each institution). The CNN model's segmentation accuracy was assessed with the Dice coefficient, and the average symmetric surface distance was assessed through fourfold cross-validation. Further, absolute difference of HU was compared between manually and automatically segmented regions. The system was tested on the remaining 1000 cases. For each institution, linear regression was applied to calculate the correlation coefficients between HU and phantom density.

RESULTS

The source code and the model used for phantom segmentation can be accessed at https://github.com/keisuke-uemura/CT-Intensity-Calibration-Phantom-Segmentation . The median Dice coefficient was 0.977, and the median average symmetric surface distance was 0.116 mm. The median absolute difference of the segmented regions between manual and automated segmentation was 0.114 HU. For the test cases, the median correlation coefficients were 0.9998 and 0.999 for the two institutions, with a minimum value of 0.9863.

CONCLUSION

The proposed CNN model successfully segmented the calibration phantom regions in CT images with excellent accuracy.

Use of artificial intelligence in the imaging of sarcopenia: A narrative review of current status and perspectives

Sarcopenia is a muscle disease which previously was associated only with aging, but in recent days it has been gaining more attention for its predictive value in a vast range of conditions and its potential link with overall health. Up to this point, evaluating sarcopenia with imaging methods has been time-consuming and dependent on the skills of the physician. The solution for this problem may be found in artificial intelligence, which may assist radiologists in repetitive tasks such as muscle segmentation and body-composition analysis. The major aim of this review was to find and present the current status and future perspectives of artificial intelligence in the imaging of sarcopenia. We searched the PubMed database to find articles concerning the use of artificial intelligence in diagnostic imaging and especially in body-composition analysis in the context of sarcopenia. We found that artificial-intelligence systems could potentially help with evaluating sarcopenia and better predicting outcomes in a vast range of clinical situations, which could get us closer to the true era of precision medicine.

Using Spatial Probability Maps to Highlight Potential Inaccuracies in Deep Learning-Based Contours: Facilitating Online Adaptive Radiation Therapy

PURPOSE

Contouring organs at risk remains a largely manual task, which is time consuming and prone to variation. Deep learning-based delineation (DLD) shows promise both in terms of quality and speed, but it does not yet perform perfectly. Because of that, manual checking of DLD is still recommended. There are currently no commercial tools to focus attention on the areas of greatest uncertainty within a DLD contour. Therefore, we explore the use of spatial probability maps (SPMs) to help efficiency and reproducibility of DLD checking and correction, using the salivary glands as the paradigm.

METHODS AND MATERIALS

A 3-dimensional fully convolutional network was trained with 315/264 parotid/submandibular glands. Subsequently, SPMs were created using Monte Carlo dropout (MCD). The method was boosted by placing a Gaussian distribution (GD) over the model's parameters during sampling (MCD + GD). MCD and MCD + GD were quantitatively compared and the SPMs were visually inspected.

RESULTS

The addition of the GD appears to increase the method's ability to detect uncertainty. In general, this technique demonstrated uncertainty in areas that (1) have lower contrast, (2) are less consistently contoured by clinicians, and (3) deviate from the anatomic norm.

CONCLUSIONS

We believe the integration of uncertainty information into contours made using DLD is an important step in highlighting where a contour may be less reliable. We have shown how SPMs are one way to achieve this and how they may be integrated into the online adaptive radiation therapy workflow.

Automated Segmentation of Abdominal Skeletal Muscle on Pediatric CT Scans Using Deep Learning

PURPOSE

To automate skeletal muscle segmentation in a pediatric population using convolutional neural networks that identify and segment the L3 level at CT.

MATERIALS AND METHODS

In this retrospective study, two sets of U-Net-based models were developed to identify the L3 level in the sagittal plane and segment the skeletal muscle from the corresponding axial image. For model development, 370 patients (sampled uniformly across age group from 0 to 18 years and including both sexes) were selected between January 2009 and January 2019, and ground truth L3 location and skeletal muscle segmentation were manually defined. Twenty percent (74 of 370) of the examinations were reserved for testing the L3 locator and muscle segmentation, while the remaining were used for training. For the L3 locator models, maximum intensity projections (MIPs) from a fixed number of central sections of sagittal reformats (either 12 or 18 sections) were used as input with or without transfer learning using an L3 localizer trained on an external dataset (four models total). For the skeletal muscle segmentation models, two loss functions (weighted Dice similarity coefficient [DSC] and binary cross-entropy) were used on models trained with or without data augmentation (four models total). Outputs from each model were compared with ground truth, and the mean relative error and DSC from each of the models were compared with one another.

RESULTS

L3 section detection trained with an 18-section MIP model with transfer learning had a mean error of 3.23 mm ± 2.61 standard deviation, which was within the reconstructed image thickness (3 or 5 mm). Skeletal muscle segmentation trained with the weighted DSC loss model without data augmentation had a mean DSC of 0.93 ± 0.03 and mean relative error of 0.04 ± 0.04.

CONCLUSION

Convolutional neural network models accurately identified the L3 level and segmented the skeletal muscle on pediatric CT scans.Supplemental material is available for this article.See also the commentary by Cadrin-Chênevert in this issue.© RSNA, 2021.

Deep Learning Enables Superior Photoacoustic Imaging at Ultralow Laser Dosages

Optical-resolution photoacoustic microscopy (OR-PAM) is an excellent modality for in vivo biomedical imaging as it noninvasively provides high-resolution morphologic and functional information without the need for exogenous contrast agents. However, the high excitation laser dosage, limited imaging speed, and imperfect image quality still hinder the use of OR-PAM in clinical applications. The laser dosage, imaging speed, and image quality are mutually restrained by each other, and thus far, no methods have been proposed to resolve this challenge. Here, a deep learning method called the multitask residual dense network is proposed to overcome this challenge. This method utilizes an innovative strategy of integrating multisupervised learning, dual-channel sample collection, and a reasonable weight distribution. The proposed deep learning method is combined with an application-targeted modified OR-PAM system. Superior images under ultralow laser dosage (32-fold reduced dosage) are obtained for the first time in this study. Using this new technique, a high-quality, high-speed OR-PAM system that meets clinical requirements is now conceivable.