Annotation of enhanced radiographs for medical image retrieval with deep convolutional neural networks

Automated location of orofacial landmarks to characterize airway morphology in anaesthesia via deep convolutional neural networks

BACKGROUND

A reliable anticipation of a difficult airway may notably enhance safety during anaesthesia. In current practice, clinicians use bedside screenings by manual measurements of patients' morphology.

OBJECTIVE

To develop and evaluate algorithms for the automated extraction of orofacial landmarks, which characterize airway morphology.

METHODS

We defined 27 frontal + 13 lateral landmarks. We collected n=317 pairs of pre-surgery photos from patients undergoing general anaesthesia (140 females, 177 males). As ground truth reference for supervised learning, landmarks were independently annotated by two anaesthesiologists. We trained two ad-hoc deep convolutional neural network architectures based on InceptionResNetV2 (IRNet) and MobileNetV2 (MNet), to predict simultaneously: (a) whether each landmark is visible or not (occluded, out of frame), (b) its 2D-coordinates (x,y). We implemented successive stages of transfer learning, combined with data augmentation. We added custom top layers on top of these networks, whose weights were fully tuned for our application. Performance in landmark extraction was evaluated by 10-fold cross-validation (CV) and compared against 5 state-of-the-art deformable models.

RESULTS

With annotators' consensus as the 'gold standard', our IRNet-based network performed comparably to humans in the frontal view: median CV loss L=1.277·10^-3, inter-quartile range (IQR) [1.001, 1.660]; versus median 1.360, IQR [1.172, 1.651], and median 1.352, IQR [1.172, 1.619], for each annotator against consensus, respectively. MNet yielded slightly worse results: median 1.471, IQR [1.139, 1.982]. In the lateral view, both networks attained performances statistically poorer than humans: median CV loss L=2.141·10^-3, IQR [1.676, 2.915], and median 2.611, IQR [1.898, 3.535], respectively; versus median 1.507, IQR [1.188, 1.988], and median 1.442, IQR [1.147, 2.010] for both annotators. However, standardized effect sizes in CV loss were small: 0.0322 and 0.0235 (non-significant) for IRNet, 0.1431 and 0.1518 (p<0.05) for MNet; therefore quantitatively similar to humans. The best performing state-of-the-art model (a deformable regularized Supervised Descent Method, SDM) behaved comparably to our DCNNs in the frontal scenario, but notoriously worse in the lateral view.

CONCLUSIONS

We successfully trained two DCNN models for the recognition of 27 + 13 orofacial landmarks pertaining to the airway. Using transfer learning and data augmentation, they were able to generalize without overfitting, reaching expert-like performances in CV. Our IRNet-based methodology achieved a satisfactory identification and location of landmarks: particularly in the frontal view, at the level of anaesthesiologists. In the lateral view, its performance decayed, although with a non-significant effect size. Independent authors had also reported lower lateral performances; as certain landmarks may not be clear salient points, even for a trained human eye.

Sociodemographic data and APOE-ε4 augmentation for MRI-based detection of amnestic mild cognitive impairment using deep learning systems

Detection and diagnosis of early and subclinical stages of Alzheimer's Disease (AD) play an essential role in the implementation of intervention and prevention strategies. Neuroimaging techniques predominantly provide insight into anatomic structure changes associated with AD. Deep learning methods have been extensively applied towards creating and evaluating models capable of differentiating between cognitively unimpaired, patients with Mild Cognitive Impairment (MCI) and AD dementia. Several published approaches apply information fusion techniques, providing ways of combining several input sources in the medical domain, which contributes to knowledge of broader and enriched quality. The aim of this paper is to fuse sociodemographic data such as age, marital status, education and gender, and genetic data (presence of an apolipoprotein E (APOE)-ε4 allele) with Magnetic Resonance Imaging (MRI) scans. This enables enriched multi-modal features, that adequately represent the MRI scan visually and is adopted for creating and modeling classification systems capable of detecting amnestic MCI (aMCI). To fully utilize the potential of deep convolutional neural networks, two extra color layers denoting contrast intensified and blurred image adaptations are virtually augmented to each MRI scan, completing the Red-Green-Blue (RGB) color channels. Deep convolutional activation features (DeCAF) are extracted from the average pooling layer of the deep learning system Inception_v3. These features from the fused MRI scans are used as visual representation for the Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) classification model. The proposed approach is evaluated on a sub-study containing 120 participants (aMCI = 61 and cognitively unimpaired = 59) of the Heinz Nixdorf Recall (HNR) Study with a baseline model accuracy of 76%. Further evaluation was conducted on the ADNI Phase 1 dataset with 624 participants (aMCI = 397 and cognitively unimpaired = 227) with a baseline model accuracy of 66.27%. Experimental results show that the proposed approach achieves 90% accuracy and 0.90 F1-Score at classification of aMCI vs. cognitively unimpaired participants on the HNR Study dataset, and 77% accuracy and 0.83 F1-Score on the ADNI dataset.

Artificial intelligence in clinical imaging: a health system approach

The development and application of artificial intelligence (AI) to radiology requires an approach that encompasses a health system. The UK government and National Health Service (NHS) are creating an ecosystem to facilitate academic/industrial partnerships aimed at accelerating the creation of relevant and robust AI tools, which will improve the development and delivery of healthcare imaging. A series of recent initiatives are described, which will drive the development and adoption of AI in clinical imaging.

Deep Convolutional Neural Networks for Automatic Detection of Orbital Blowout Fractures

Orbital blow out fracture is a common disease in emergency department and a delay or failure in diagnosis can lead to permanent visual changes. This study aims to evaluate the ability of an automatic orbital blowout fractures detection system based on computed tomography (CT) data.Orbital CT scans of adult orbital blowout fractures patients and normal cases were obtained from Shanghai Ninth People's Hospital between January and March 2017. The region of fractures was annotated using 3D Slicer. The Inception V3 convolutional neural networks were constructed utilizing the Python programming language with PyTorch as the framework to extract high dimension features from each slice in a CT scan. These extracted features are processed through a XGBoost model to make the final differentiation of fracture cases and nonfracture ones. Accuracy, receiver operating characteristics, and area under the curve were evaluated.This study used 94 CT scans diagnosed with orbital blowout fractures and 94 healthy control cases. The automatic detection system showed accuracy of 92% in single-image classification and 87% in patient level detection. The area under the receiver operating characteristic curve was 0.9574.Using a deep learning-based automatic detection system of orbital blowout fracture can accurately detect and classify orbital blowout fractures from CT scans. The convolutional neural networks model combined with an accurate annotation system could achieve good performance in a small dataset. Further studies with large and multicenter data are required to refine this technology for possible clinical applications.