Content-based image retrieval in radiology: analysis of variability in human perception of similarity

Saliency Inside: Learning Attentive CNNs for Content-based Image Retrieval

In content-based image retrieval (CBIR), one of the most challenging and ambiguous tasks are to correctly understand the human query intention and measure its semantic relevance with images in the database. Due to the impressive capability of visual saliency in predicting human visual attention that is closely related to the query intention, this paper attempts to explicitly discover the essential effect of visual saliency in CBIR via qualitative and quantitative experiments. Toward this end, we first generate the fixation density maps of images from a widely used CBIR dataset by using an eye-tracking apparatus. These ground-truth saliency maps are then used to measure the influence of visual saliency to the task of CBIR by exploring several probable ways of incorporating such saliency cues into the retrieval process. We find that visual saliency is indeed beneficial to the CBIR task, and the best saliency involving scheme is possibly different for different image retrieval models. Inspired by the findings, this paper presents two-stream attentive CNNs with saliency embedded inside for CBIR. The proposed network has two streams that simultaneously handle two tasks. The main stream focuses on extracting discriminative visual features that are tightly related to semantic attributes. Meanwhile, the auxiliary stream aims to facilitate the main stream by redirecting the feature extraction to the salient image content that human may pay attention to. By fusing these two streams into the Main and Auxiliary CNNs (MAC), image similarity can be computed as the human being does by reserving conspicuous content and suppressing irrelevant regions. Extensive experiments show that the proposed model achieves impressive performance in image retrieval on four public datasets.

Automated Anatomic Labeling Architecture for Content Discovery in Medical Imaging Repositories

The combination of textual data with visual features is known to enhance medical image search capabilities. However, the most advanced imaging archives today only index the studies' available meta-data, often containing limited amounts of clinically useful information. This work proposes an anatomic labeling architecture, integrated with an open source archive software, for improved multimodal content discovery in real-world medical imaging repositories. The proposed solution includes a technical specification for classifiers in an extensible medical imaging archive, a classification database for querying over the extracted information, and a set of proof-of-concept convolutional neural network classifiers for identifying the presence of organs in computed tomography scans. The system automatically extracts the anatomic region features, which are saved in the proposed database for later consumption by multimodal querying mechanisms. The classifiers were evaluated with cross-validation, yielding a best F1-score of 96% and an average accuracy of 97%. We expect these capabilities to become common-place in production environments in the future, as automated detection solutions improve in terms of accuracy, computational performance, and interoperability.

A Multimodal Search Engine for Medical Imaging Studies

The use of digital medical imaging systems in healthcare institutions has increased significantly, and the large amounts of data in these systems have led to the conception of powerful support tools: recent studies on content-based image retrieval (CBIR) and multimodal information retrieval in the field hold great potential in decision support, as well as for addressing multiple challenges in healthcare systems, such as computer-aided diagnosis (CAD). However, the subject is still under heavy research, and very few solutions have become part of Picture Archiving and Communication Systems (PACS) in hospitals and clinics. This paper proposes an extensible platform for multimodal medical image retrieval, integrated in an open-source PACS software with profile-based CBIR capabilities. In this article, we detail a technical approach to the problem by describing its main architecture and each sub-component, as well as the available web interfaces and the multimodal query techniques applied. Finally, we assess our implementation of the engine with computational performance benchmarks.

Systems Medicine: The Application of Systems Biology Approaches for Modern Medical Research and Drug Development

The exponential development of highly advanced scientific and medical research technologies throughout the past 30 years has arrived to the point where the high number of characterized molecular agents related to pathogenesis cannot be readily integrated or processed by conventional analytical approaches. Indeed, the realization that several moieties are signatures of disease has partly led to the increment of complex diseases being characterized. Scientists and clinicians can now investigate and analyse any individual dysregulations occurring within the genomic, transcriptomic, miRnomic, proteomic, and metabolomic levels thanks to currently available advanced technologies. However, there are drawbacks within this scientific brave new age in that only isolated molecular levels are individually investigated for their influence in affecting any particular health condition. Since their conception in 1992, systems biology/medicine focuses mainly on the perturbations of overall pathway kinetics for the consequent onset and/or deterioration of the investigated condition/s. Systems medicine approaches can therefore be employed for shedding light in multiple research scenarios, ultimately leading to the practical result of uncovering novel dynamic interaction networks that are critical for influencing the course of medical conditions. Consequently, systems medicine also serves to identify clinically important molecular targets for diagnostic and therapeutic measures against such a condition.