Generating diagnostic report for medical image by high-middle-level visual information incorporation on double deep learning models

From vision to text: A comprehensive review of natural image captioning in medical diagnosis and radiology report generation

Natural Image Captioning (NIC) is an interdisciplinary research area that lies within the intersection of Computer Vision (CV) and Natural Language Processing (NLP). Several works have been presented on the subject, ranging from the early template-based approaches to the more recent deep learning-based methods. This paper conducts a survey in the area of NIC, especially focusing on its applications for Medical Image Captioning (MIC) and Diagnostic Captioning (DC) in the field of radiology. A review of the state-of-the-art is conducted summarizing key research works in NIC and DC to provide a wide overview on the subject. These works include existing NIC and MIC models, datasets, evaluation metrics, and previous reviews in the specialized literature. The revised work is thoroughly analyzed and discussed, highlighting the limitations of existing approaches and their potential implications in real clinical practice. Similarly, future potential research lines are outlined on the basis of the detected limitations.

Improving preliminary clinical diagnosis accuracy through knowledge filtering techniques in consultation dialogues

BACKGROUND AND OBJECTIVE

Symptom descriptions by ordinary people are often inaccurate or vague when seeking medical advice, which often leads to inaccurate preliminary clinical diagnoses. To address this issue, we propose a deep learning model named the knowledgeable diagnostic transformer (KDT) for the natural language processing (NLP)-based preliminary clinical diagnoses.

METHODS

The KDT extracts symptom-disease relation triples (h,r,t) from patient symptom descriptions by using a proposed bipartite medical knowledge graph (bMKG). To avoid too many relation triples causing the knowledge noise issue, we propose a knowledge inclusion-exclusion approach (KIA) to eliminate undesirable triples (a knowledge filtering layer). Next, we combine token embedding techniques with the transformer model to predict the diseases that patients may encounter.

RESULTS

To train the KDT, a medical diagnosis question-answering dataset (named MDQA dataset) containing large-scale, high-quality questions (patient syndrome description) and answering (diagnosis) corpora with 2.6M entries (1.07GB in size) in Mandarin was built. We also train the KDT with the National Institutes of Health (NIH) English dataset (MedQuAD). The KDT marks a transformative approach by achieving a remarkable accuracy of 99% for different evaluation metrics when compared with the baseline transformers used for the NLP-based preliminary clinical diagnoses approaches.

CONCLUSIONS

In essence, our study not only demonstrates the effectiveness of the KDT in enhancing diagnostic precision but also underscores its potential to revolutionize the field of preliminary clinical diagnoses. By harnessing the power of knowledge-based approaches and advanced NLP techniques, we have paved the way for more accurate and reliable diagnoses, ultimately benefiting both healthcare providers and patients. The KDT has the potential to significantly reduce misdiagnoses and improve patient outcomes, marking a pivotal advancement in the realm of medical diagnostics.

AERMNet: Attention-enhanced relational memory network for medical image report generation

BACKGROUND AND OBJECTIVES

The automatic generation of medical image diagnostic reports can assist doctors in reducing their workload and improving the efficiency and accuracy of diagnosis. However, among the most existing report generation models, there are problems that the weak correlation between generated words and the lack of contextual information in the report generation process.

METHODS

To address the above problems, we propose an Attention-Enhanced Relational Memory Network (AERMNet) model, where the relational memory module is continuously updated by the words generated in the previous time step to strengthen the correlation between words in generated medical image report. And the double LSTM with interaction module reduces the loss of context information and makes full use of feature information. Thus, more accurate disease information can be generated by AERMNet for medical image reports.

RESULTS

Experimental results on four medical datasets Fetal heart (FH), Ultrasound, IU X-Ray and MIMIC-CXR, show that our proposed method outperforms some of the previous models with respect to language generation metrics (Cider improving by 2.4% on FH, Bleu1 improving by 2.4% on Ultrasound, Cider improving by 16.4% on IU X-Ray, Bleu2 improving by 9.7% on MIMIC-CXR).

CONCLUSIONS

This work promotes the development of medical image report generation and expands the prospects of computer-aided diagnosis applications. Our code is released at https://github.com/llttxx/AERMNET.

Translating medical image to radiological report: Adaptive multilevel multi-attention approach

BACKGROUND AND OBJECTIVE

Medical imaging techniques are widely employed in disease diagnosis and treatment. A readily available medical report can be a useful tool in assisting an expert for investigating the patient's health. A radiologist can benefit from an automatic medical image to radiological report translation system while preparing a final report. Previous attempts on automatic medical report generation task includes image captioning algorithms without taking domain-specific visual and textual contents into account, thus arises the question about credibility of generated report.

METHODS

In this work, a novel Adaptive Multilevel Multi-Attention (AMLMA) approach is proposed by offering domain-specific visual-textual knowledge to generate a thorough and believable radiological report for any view of a human chest X-ray image. The proposed approach leverages the encoder-decoder framework incorporated with multiple adaptive attention mechanisms. The potential of a convolutional neural network (CNN) with residual attention module (RAM) is demonstrated as a strong visual encoder for multi-label abnormality detection. The multilevel visual features (local and global) are extracted from proposed visual encoder to retrieve regional-level and abstract-level radiology-based semantic information. The Word2Vec and FastText word embeddings are trained on medical reports to acquire radiological knowledge and further used as textual encoders, feeding as input to Bi-directional Long Short Term Memory (Bi-LSTM) network to learn the co-relationship between medical terminologies in radiological reports. The AMLMA employs a weighted multilevel association of adaptive visual-semantic attention and visual-based linguistic attention mechanisms. This association of adaptive attention is exploited as a decoder and produces significant improvements in the report generation task.

RESULTS

The proposed approach is evaluated on a publicly available Indiana University chest X-ray (IU-CXR) dataset. The CNN with RAM shows the significant improvement in recall (0.4423), precision (0.1803) and F1-score (0.2551) for prediction of multiple abnormalities in X-ray image. The results of language generation metrics for proposed variants were acquired using the COCO-caption evaluation Application Program Interface (API). The trained embeddings with AMLMA model generates the convincing radiology report and outperform state-of-the-art (SOTA) approaches with high evaluation metrics scores for Bleu-4 (0.172), Meteor (0.247), Rouge_L (0.376) and CIDEr (0.381). In addition, a new "Unique Index" (UI) statistic is introduced to highlight the model's ability for generating unique reports.

CONCLUSION

The overall architecture aids to the understanding of various X-ray image views and generating the relevant normal and abnormal radiography statements. The proposed model is emphasized on multi-level visual-textual knowledge with adaptive attention mechanism to balance visual and linguistic information for the generation of admissible radiology report.

Ultrasonic Image Feature Analysis under Deep Learning Algorithm to Evaluate the Efficacy of Drug-Coated Balloon for Treatment of Arteriosclerotic Occlusion

In order to explore the adoption of ultrasonic images under deep learning (DL) algorithm to evaluate the efficacy of drug-coated balloon (DCB) for treatment of arteriosclerotic occlusion, 56 patients who underwent DCB surgery of lower limb artery were selected and all the patients received the examinations of algorithmic ultrasound and digital subtraction angiography (DSA) before surgery. According to the examination methods, they were classified into algorithmic ultrasound group and DSA group. One to two months after DCB surgery, ultrasound examination was performed with the region-based faster convolutional neural network (faster R-CNN) target detection algorithm to check the therapeutic effect. The results showed that the image effect processed by the target detection algorithm based on DL was signally better than that of traditional ultrasonic processing algorithm in Dice, precision (Pre), and sensitivity, with significant difference (P < 0.05). Compared with DSA, algorithmic ultrasound showed better consistency between the two groups in the diagnosis of common femoral artery, superficial femoral artery, and popliteal artery stenosis, with statistical significance (P < 0.05). However, for the diagnosis of anterior tibial artery stenosis, the consistency between algorithmic ultrasound and DSA was general. The residual stenosis of each artery segment decreased obviously in postoperative review compared with that before surgery, and the difference was statistically significant (P < 0.05). Besides, both the pulsatility index (PI) and the blood flow velocity of the dorsalis pedis artery increased after surgery, compared with those before surgery, with significant differences (P < 0.05). To sum up, ultrasound based on DL target detection algorithm had good imaging effect and good consistency with DSA, which was of the clinical reference value. Additionally, DCB surgery was helpful to treat arteriosclerosis occlusion and improve limb blood supply, which had clinical adoption value.