A Knowledge Distillation Ensemble Framework for Predicting Short- and Long-Term Hospitalization Outcomes From Electronic Health Records Data

Leveraging different learning styles for improved knowledge distillation in biomedical imaging

Learning style refers to a type of training mechanism adopted by an individual to gain new knowledge. As suggested by the VARK model, humans have different learning preferences, like Visual (V), Auditory (A), Read/Write (R), and Kinesthetic (K), for acquiring and effectively processing information. Our work endeavors to leverage this concept of knowledge diversification to improve the performance of model compression techniques like Knowledge Distillation (KD) and Mutual Learning (ML). Consequently, we use a single-teacher and two-student network in a unified framework that not only allows for the transfer of knowledge from teacher to students (KD) but also encourages collaborative learning between students (ML). Unlike the conventional approach, where the teacher shares the same knowledge in the form of predictions or feature representations with the student network, our proposed approach employs a more diversified strategy by training one student with predictions and the other with feature maps from the teacher. We further extend this knowledge diversification by facilitating the exchange of predictions and feature maps between the two student networks, enriching their learning experiences. We have conducted comprehensive experiments with three benchmark datasets for both classification and segmentation tasks using two different network architecture combinations. These experimental results demonstrate that knowledge diversification in a combined KD and ML framework outperforms conventional KD or ML techniques (with similar network configuration) that only use predictions with an average improvement of 2%. Furthermore, consistent improvement in performance across different tasks, with various network architectures, and over state-of-the-art techniques establishes the robustness and generalizability of the proposed model.

Subgroup State Prediction under Different Noise Levels Using MODWT and XGBoost

In medical states prediction, the observations of different individuals are generally assumed to follow an identical distribution, whereas precision medicine has a rigorous requirement for accurate subgroup analysis. In this research, an aggregated method is proposed by means of combining the results generated from different subgroup models and is compared with the original method for different denoising levels as well as the prediction gaps. The results using real data demonstrate the effectiveness of the aggregated method exhibiting superior performance such as 0.95 in AUC, 0.87 in F1, and 0.82 in sensitivity, particularly for the denoising level that is set to be 2. With respect to the variable importance, it is shown that some variables such as heart rate and lactate arterial become more important when the denoising level increases.