Pay attention to doctor-patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis

Identification of mycoplasma pneumonia in children based on fusion of multi-modal clinical free-text description and structured test data

Mycoplasma pneumonia may lead to hospitalizations and pose life-threatening risks in children. The automated identification of mycoplasma pneumonia from electronic medical records holds significant potential for improving the efficiency of hospital resource allocation. In this study, we proposed a novel method for identifying mycoplasma pneumonia by integrating multi-modal features derived from both free-text descriptions and structured test data in electronic medical records. Our approach begins with the extraction of free-text and structured data from clinical records through a systematic preprocessing pipeline. Subsequently, we employ a pre-trained transformer language model to extract features from the free-text, while multiple additive regression trees are used to transform features from the structured data. An attention-based fusion mechanism is then applied to integrate these multi-modal features for effective classification. We validated our method using clinic records of 7157 patients, retrospectively collected for training and testing purposes. The experimental results demonstrate that our proposed multi-modal fusion approach achieves significant improvements over other methods across four key performance metrics.

Music recommendation algorithms based on knowledge graph and multi-task feature learning

During music recommendation scenarios, sparsity and cold start problems are inevitable. Auxiliary information has been utilized in music recommendation algorithms to provide users with more accurate music recommendation results. This study proposes an end-to-end framework, MMSS_MKR, that uses a knowledge graph as a source of auxiliary information to serve the information obtained from it to the recommendation module. The framework exploits Cross & Compression Units to bridge the knowledge graph embedding task with recommendation task modules. We can obtain more realistic triple information and exclude false triple information as much as possible, because our model obtains triple information through the music knowledge graph, and the information obtained through the recommendation module is used to determine the truth of the triple information; thus, the knowledge graph embedding task is used to perform the recommendation task. In the recommendation module, multiple predictions are adopted to predict the recommendation accuracy. In the knowledge graph embedding module, multiple calculations are used to calculate the score. Finally, the loss function of the model is improved to help us to obtain more useful information for music recommendations. The MMSS_MKR model achieved significant improvements in music recommendations compared with many existing recommendation models.

A scoping review on multimodal deep learning in biomedical images and texts

OBJECTIVE

Computer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers' attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions.

METHODS

In this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research.

RESULT

This study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation.

CONCLUSION

Our results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development.

EE-Explorer: A Multimodal Artificial Intelligence System for Eye Emergency Triage and Primary Diagnosis

PURPOSE

To develop a multimodal artificial intelligence (AI) system, EE-Explorer, to triage eye emergencies and assist in primary diagnosis using metadata and ocular images.

DESIGN

A diagnostic, cross-sectional, validity and reliability study.

METHODS

EE-Explorer consists of 2 models. The triage model was developed from metadata (events, symptoms, and medical history) and ocular surface images via smartphones from 2038 patients presenting to Zhongshan Ophthalmic Center (ZOC) to output 3 classifications: urgent, semiurgent, and nonurgent. The primary diagnostic model was developed from the paired metadata and slitlamp images of 2405 patients from ZOC. Both models were externally tested on 103 participants from 4 other hospitals. A pilot test was conducted in Guangzhou to evaluate the hierarchical referral service pattern assisted by EE-Explorer for unspecialized health care facilities.

RESULTS

A high overall accuracy, as indicated by an area under the receiver operating characteristic curve (AUC) of 0.982 (95% CI, 0.966-0.998), was obtained using the triage model, which outperformed the triage nurses (P < .001). In the primary diagnostic model, the diagnostic classification accuracy (CA) and Hamming loss (HL) in the internal testing were 0.808 (95% CI 0.776-0.840) and 0.016 (95% CI 0.006-0.026), respectively. In the external testing, model performance was robust for both triage (average AUC, 0.988, 95% CI 0.967-1.000) and primary diagnosis (CA, 0.718, 95% CI 0.644-0.792; and HL, 0.023, 95% CI 0.000-0.048). In the pilot test in the hierarchical referral settings, EE-explorer demonstrated consistently robust performance and broad participant acceptance.

CONCLUSION

The EE-Explorer system showed robust performance in both triage and primary diagnosis for ophthalmic emergency patients. EE-Explorer can provide patients with acute ophthalmic symptoms access to remote self-triage and assist in primary diagnosis in unspecialized health care facilities to achieve rapid and effective treatment strategies.

Structural Attention Graph Neural Network for Diagnosis and Prediction of COVID-19 Severity

With rapid worldwide spread of Coronavirus Disease 2019 (COVID-19), jointly identifying severe COVID-19 cases from mild ones and predicting the conversion time (from mild to severe) is essential to optimize the workflow and reduce the clinician's workload. In this study, we propose a novel framework for COVID-19 diagnosis, termed as Structural Attention Graph Neural Network (SAGNN), which can combine the multi-source information including features extracted from chest CT, latent lung structural distribution, and non-imaging patient information to conduct diagnosis of COVID-19 severity and predict the conversion time from mild to severe. Specifically, we first construct a graph to incorporate structural information of the lung and adopt graph attention network to iteratively update representations of lung segments. To distinguish different infection degrees of left and right lungs, we further introduce a structural attention mechanism. Finally, we introduce demographic information and develop a multi-task learning framework to jointly perform both tasks of classification and regression. Experiments are conducted on a real dataset with 1687 chest CT scans, which includes 1328 mild cases and 359 severe cases. Experimental results show that our method achieves the best classification (e.g., 86.86% in terms of Area Under Curve) and regression (e.g., 0.58 in terms of Correlation Coefficient) performance, compared with other comparison methods.

An ACP-Based Parallel Approach for Color Image Encryption Using Redundant Blocks

Public concerns on image encryption grow significantly as the development and application of edge computing and the Internet of Things intensified recently. However, most existing image cryptosystems are not sophisticated enough to resist the two major attack strategies available currently, that is: 1) differential attacks and 2) chosen-plaintext attacks, which are famous for their destructive power, especially their capability of exploiting cryptosystems' features to recover the secret key. In this article, we propose an artificial image, computational experiment, and parallel execution (ACP)-based color image encryption approach using redundant blocks. First, a redundant blocks strategy with redundant spaces is proposed to prevent differential attacks and accelerate operating speed while guaranteeing the security of image cryptosystems. Second, real-world chaotic data (e.g., stock data) are obtained to generate artificial images and conduct computational experiments. Furthermore, artificial images are encrypted via real-world chaos, while the original images are encrypted via simulated chaos (such as Chen's hyperchaos). Finally, we design the process of parallel execution for image encryption and use DNA XOR to merge two groups of encrypted subimages to fuse the effect of the chaotic characteristics in both the real world and the simulation. The final encrypted image is realized through the recovery of redundant blocks. The ACP mechanism of color image encryption achieves the goal of improving the sophistication of chaos-based cryptosystems and resists both the differential and chosen-plaintext attacks. Experimental results and security analysis show that our approach provides not only excellent encryption but also security sufficient to prevent known attacks.

Knowledge-Based Recurrent Neural Network for TCM Cerebral Palsy Diagnosis

Cerebral palsy is one of the most prevalent neurological disorders and the most frequent cause of disability. Identifying the syndrome by patients' symptoms is the key to traditional Chinese medicine (TCM) cerebral palsy treatment. Artificial intelligence (AI) is advancing quickly in several sectors, including TCM. AI will considerably enhance the dependability and precision of diagnoses, expanding effective treatment methods' usage. Thus, for cerebral palsy, it is necessary to build a decision-making model to aid in the syndrome diagnosis process. While the recurrent neural network (RNN) model has the potential to capture the correlation between symptoms and syndromes from electronic medical records (EMRs), it lacks TCM knowledge. To make the model benefit from both TCM knowledge and EMRs, unlike the ordinary training routine, we begin by constructing a knowledge-based RNN (KBRNN) based on the cerebral palsy knowledge graph for domain knowledge. More specifically, we design an evolution algorithm for extracting knowledge in the cerebral palsy knowledge graph. Then, we embed the knowledge into tensors and inject them into the RNN. In addition, the KBRNN can benefit from the labeled EMRs. We use EMRs to fine-tune the KBRNN, which improves prediction accuracy. Our study shows that knowledge injection can effectively improve the model effect. The KBRNN can achieve 79.31% diagnostic accuracy with only knowledge injection. Moreover, the KBRNN can be further trained by the EMRs. The results show that the accuracy of fully trained KBRNN is 83.12%.

Comparison of Convolutional Neural Networks and Transformers for the Classification of Images of COVID-19, Pneumonia and Healthy Individuals as Observed with Computed Tomography

In this work, the performance of five deep learning architectures in classifying COVID-19 in a multi-class set-up is evaluated. The classifiers were built on pretrained ResNet-50, ResNet-50r (with kernel size 5×5 in the first convolutional layer), DenseNet-121, MobileNet-v3 and the state-of-the-art CaiT-24-XXS-224 (CaiT) transformer. The cross entropy and weighted cross entropy were minimised with Adam and AdamW. In total, 20 experiments were conducted with 10 repetitions and obtained the following metrics: accuracy (Acc), balanced accuracy (BA), F₁ and F₂ from the general Fβ macro score, Matthew’s Correlation Coefficient (MCC), sensitivity (Sens) and specificity (Spec) followed by bootstrapping. The performance of the classifiers was compared by using the Friedman–Nemenyi test. The results show that less complex architectures such as ResNet-50, ResNet-50r and DenseNet-121 were able to achieve better generalization with rankings of 1.53, 1.71 and 3.05 for the Matthew Correlation Coefficient, respectively, while MobileNet-v3 and CaiT obtained rankings of 3.72 and 5.0, respectively.

Knowledge Graph Applications in Medical Imaging Analysis: A Scoping Review

Background

There is an increasing trend to represent domain knowledge in structured graphs, which provide efficient knowledge representations for many downstream tasks. Knowledge graphs are widely used to model prior knowledge in the form of nodes and edges to represent semantically connected knowledge entities, which several works have adopted into different medical imaging applications.

Methods

We systematically searched over five databases to find relevant articles that applied knowledge graphs to medical imaging analysis. After screening, evaluating, and reviewing the selected articles, we performed a systematic analysis.

Results

We looked at four applications in medical imaging analysis, including disease classification, disease localization and segmentation, report generation, and image retrieval. We also identified limitations of current work, such as the limited amount of available annotated data and weak generalizability to other tasks. We further identified the potential future directions according to the identified limitations, including employing semisupervised frameworks to alleviate the need for annotated data and exploring task-agnostic models to provide better generalizability.

Conclusions

We hope that our article will provide the readers with aggregated documentation of the state-of-the-art knowledge graph applications for medical imaging to encourage future research.

Review of Machine Learning in Lung Ultrasound in COVID-19 Pandemic

Ultrasound imaging of the lung has played an important role in managing patients with COVID-19–associated pneumonia and acute respiratory distress syndrome (ARDS). During the COVID-19 pandemic, lung ultrasound (LUS) or point-of-care ultrasound (POCUS) has been a popular diagnostic tool due to its unique imaging capability and logistical advantages over chest X-ray and CT. Pneumonia/ARDS is associated with the sonographic appearances of pleural line irregularities and B-line artefacts, which are caused by interstitial thickening and inflammation, and increase in number with severity. Artificial intelligence (AI), particularly machine learning, is increasingly used as a critical tool that assists clinicians in LUS image reading and COVID-19 decision making. We conducted a systematic review from academic databases (PubMed and Google Scholar) and preprints on arXiv or TechRxiv of the state-of-the-art machine learning technologies for LUS images in COVID-19 diagnosis. Openly accessible LUS datasets are listed. Various machine learning architectures have been employed to evaluate LUS and showed high performance. This paper will summarize the current development of AI for COVID-19 management and the outlook for emerging trends of combining AI-based LUS with robotics, telehealth, and other techniques.