Mixed reality assists the fight against COVID-19

A modality-collaborative convolution and transformer hybrid network for unpaired multi-modal medical image segmentation with limited annotations

BACKGROUND

Multi-modal learning is widely adopted to learn the latent complementary information between different modalities in multi-modal medical image segmentation tasks. Nevertheless, the traditional multi-modal learning methods require spatially well-aligned and paired multi-modal images for supervised training, which cannot leverage unpaired multi-modal images with spatial misalignment and modality discrepancy. For training accurate multi-modal segmentation networks using easily accessible and low-cost unpaired multi-modal images in clinical practice, unpaired multi-modal learning has received comprehensive attention recently.

PURPOSE

Existing unpaired multi-modal learning methods usually focus on the intensity distribution gap but ignore the scale variation problem between different modalities. Besides, within existing methods, shared convolutional kernels are frequently employed to capture common patterns in all modalities, but they are typically inefficient at learning global contextual information. On the other hand, existing methods highly rely on a large number of labeled unpaired multi-modal scans for training, which ignores the practical scenario when labeled data is limited. To solve the above problems, we propose a modality-collaborative convolution and transformer hybrid network (MCTHNet) using semi-supervised learning for unpaired multi-modal segmentation with limited annotations, which not only collaboratively learns modality-specific and modality-invariant representations, but also could automatically leverage extensive unlabeled scans for improving performance.

METHODS

We make three main contributions to the proposed method. First, to alleviate the intensity distribution gap and scale variation problems across modalities, we develop a modality-specific scale-aware convolution (MSSC) module that can adaptively adjust the receptive field sizes and feature normalization parameters according to the input. Secondly, we propose a modality-invariant vision transformer (MIViT) module as the shared bottleneck layer for all modalities, which implicitly incorporates convolution-like local operations with the global processing of transformers for learning generalizable modality-invariant representations. Third, we design a multi-modal cross pseudo supervision (MCPS) method for semi-supervised learning, which enforces the consistency between the pseudo segmentation maps generated by two perturbed networks to acquire abundant annotation information from unlabeled unpaired multi-modal scans.

RESULTS

Extensive experiments are performed on two unpaired CT and MR segmentation datasets, including a cardiac substructure dataset derived from the MMWHS-2017 dataset and an abdominal multi-organ dataset consisting of the BTCV and CHAOS datasets. Experiment results show that our proposed method significantly outperforms other existing state-of-the-art methods under various labeling ratios, and achieves a comparable segmentation performance close to single-modal methods with fully labeled data by only leveraging a small portion of labeled data. Specifically, when the labeling ratio is 25%, our proposed method achieves overall mean DSC values of 78.56% and 76.18% in cardiac and abdominal segmentation, respectively, which significantly improves the average DSC value of two tasks by 12.84% compared to single-modal U-Net models.

CONCLUSIONS

Our proposed method is beneficial for reducing the annotation burden of unpaired multi-modal medical images in clinical applications.

Artificial intelligence to detect the femoral intertrochanteric fracture: The arrival of the intelligent-medicine era

Objective: To explore a new artificial intelligence (AI)-aided method to assist the clinical diagnosis of femoral intertrochanteric fracture (FIF), and further compare the performance with human level to confirm the effect and feasibility of the AI algorithm.

Methods: 700 X-rays of FIF were collected and labeled by two senior orthopedic physicians to set up the database, 643 for the training database and 57 for the test database. A Faster-RCNN algorithm was applied to be trained and detect the FIF on X-rays. The performance of the AI algorithm such as accuracy, sensitivity, miss diagnosis rate, specificity, misdiagnosis rate, and time consumption was calculated and compared with that of orthopedic attending physicians.

Results: Compared with orthopedic attending physicians, the Faster-RCNN algorithm performed better in accuracy (0.88 vs. 0.84 ± 0.04), specificity (0.87 vs. 0.71 ± 0.08), misdiagnosis rate (0.13 vs. 0.29 ± 0.08), and time consumption (5 min vs. 18.20 ± 1.92 min). As for the sensitivity and missed diagnosis rate, there was no statistical difference between the AI and orthopedic attending physicians (0.89 vs. 0.87 ± 0.03 and 0.11 vs. 0.13 ± 0.03).

Conclusion: The AI diagnostic algorithm is an available and effective method for the clinical diagnosis of FIF. It could serve as a satisfying clinical assistant for orthopedic physicians.

Examining the effectiveness of virtual, augmented, and mixed reality (VAMR) therapy for upper limb recovery and activities of daily living in stroke patients: a systematic review and meta-analysis

INTRODUCTION

Virtual reality (VR), augmented reality (AR), and mixed reality (MR) are emerging technologies in the field of stroke rehabilitation that have the potential to overcome the limitations of conventional treatment. Enhancing upper limb (UL) function is critical in stroke impairments because the upper limb is involved in the majority of activities of daily living (ADL).

METHODS

This study reviewed the use of virtual, augmented and mixed reality (VAMR) methods for improving UL recovery and ADL, and compared the effectiveness of VAMR treatment to conventional rehabilitation therapy. The databases ScienceDirect, PubMed, IEEE Xplore, and Web of Science were examined, and 50 randomized control trials comparing VAMR treatment to standard therapy were determined. The random effect model and fixed effect model are applied based on heterogeneity.

RESULTS

The most often used outcomes of UL recovery and ADL in stroke rehabilitation were the Fugl-Meyer Assessment for Upper Extremities (FMA-UE), followed by the Box and Block Test (BBT), the Wolf Motor Function Test (WMFT), and the Functional Independence Measure (FIM). According to the meta-analysis, VR, AR, and MR all have a significant positive effect on improving FMA-UE for UL impairment (36 studies, MD = 3.91, 95 percent CI = 1.70-6.12, P = 0.0005) and FIM for ADL (10 studies, MD = 4.25, 95 percent CI = 1.47-7.03, P = 0.003), but not on BBT and WMFT for the UL function tests (16 studies, MD = 2.07, 95 percent CI = - 0.58-4.72, P = 0.13), CONCLUSIONS: VAMR therapy was superior to conventional treatment in UL impairment and daily function outcomes, but not UL function measures. Future studies might include further high-quality trials examining the effect of VR, AR, and MR on UL function measures, with an emphasis on subgroup meta-analysis by stroke type and recovery stage.