Accuracy of MRI Classification Algorithms in a Tertiary Memory Center Clinical Routine Cohort

The use of an anterior-posterior atrophy index to distinguish Alzheimer's disease from frontotemporal disorders: an automated volumetric MRI Study

BACKGROUND

Alzheimer's disease (AD) and frontotemporal dementia (FTD) require different treatments. Since clinical presentation can be nuanced, imaging biomarkers aid in diagnosis. Automated software such as Neuroreader (NR) provides volumetric imaging data, and indices between anterior and posterior brain areas have proven useful in distinguishing dementia subtypes in research cohorts. Existing indices are complex and require further validation in clinical settings.

PURPOSE

To provide initial validation for a simplified anterior-posterior index (API) from NR in distinguishing FTD and AD in a clinical cohort.

MATERIAL AND METHODS

A retrospective chart review was completed. We derived a simplified API: API = (logVA/VP-μ)/σ where V A is weighted volume of frontal and temporal lobes and V P of parietal and occipital lobes. μ and σ are the mean and standard deviation of logVA/VP computed for AD participants. Receiver operating characteristic (ROC) curves and regression analyses assessed the efficacy of the API versus brain areas in predicting diagnosis of AD versus FTD.

RESULTS

A total of 39 participants with FTD and 78 participants with AD were included. The API had an excellent performance in distinguishing AD from FTD with an area under the ROC curve of 0.82 and a positive association with diagnostic classification on logistic regression analysis (B = 1.491, P < 0.001).

CONCLUSION

The API successfully distinguished AD and FTD with excellent performance. The results provide preliminary validation of the API in a clinical setting.

Contrast-enhanced to non-contrast-enhanced image translation to exploit a clinical data warehouse of T1-weighted brain MRI

BACKGROUND

Clinical data warehouses provide access to massive amounts of medical images, but these images are often heterogeneous. They can for instance include images acquired both with or without the injection of a gadolinium-based contrast agent. Harmonizing such data sets is thus fundamental to guarantee unbiased results, for example when performing differential diagnosis. Furthermore, classical neuroimaging software tools for feature extraction are typically applied only to images without gadolinium. The objective of this work is to evaluate how image translation can be useful to exploit a highly heterogeneous data set containing both contrast-enhanced and non-contrast-enhanced images from a clinical data warehouse.

METHODS

We propose and compare different 3D U-Net and conditional GAN models to convert contrast-enhanced T1-weighted (T1ce) into non-contrast-enhanced (T1nce) brain MRI. These models were trained using 230 image pairs and tested on 77 image pairs from the clinical data warehouse of the Greater Paris area.

RESULTS

Validation using standard image similarity measures demonstrated that the similarity between real and synthetic T1nce images was higher than between real T1nce and T1ce images for all the models compared. The best performing models were further validated on a segmentation task. We showed that tissue volumes extracted from synthetic T1nce images were closer to those of real T1nce images than volumes extracted from T1ce images.

CONCLUSION

We showed that deep learning models initially developed with research quality data could synthesize T1nce from T1ce images of clinical quality and that reliable features could be extracted from the synthetic images, thus demonstrating the ability of such methods to help exploit a data set coming from a clinical data warehouse.

Evaluation of MRI-based machine learning approaches for computer-aided diagnosis of dementia in a clinical data warehouse

A variety of algorithms have been proposed for computer-aided diagnosis of dementia from anatomical brain MRI. These approaches achieve high accuracy when applied to research data sets but their performance on real-life clinical routine data has not been evaluated yet. The aim of this work was to study the performance of such approaches on clinical routine data, based on a hospital data warehouse, and to compare the results to those obtained on a research data set. The clinical data set was extracted from the hospital data warehouse of the Greater Paris area, which includes 39 different hospitals. The research set was composed of data from the Alzheimer's Disease Neuroimaging Initiative data set. In the clinical set, the population of interest was identified by exploiting the diagnostic codes from the 10th revision of the International Classification of Diseases that are assigned to each patient. We studied how the imbalance of the training sets, in terms of contrast agent injection and image quality, may bias the results. We demonstrated that computer-aided diagnosis performance was strongly biased upwards (over 17 percent points of balanced accuracy) by the confounders of image quality and contrast agent injection, a phenomenon known as the Clever Hans effect or shortcut learning. When these biases were removed, the performance was very poor. In any case, the performance was considerably lower than on the research data set. Our study highlights that there are still considerable challenges for translating dementia computer-aided diagnosis systems to clinical routine.

FDA-approved machine learning algorithms in neuroradiology: A systematic review of the current evidence for approval

Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as "medical devices" to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.

Harnessing the potential of machine learning and artificial intelligence for dementia research

Progress in dementia research has been limited, with substantial gaps in our knowledge of targets for prevention, mechanisms for disease progression, and disease-modifying treatments. The growing availability of multimodal data sets opens possibilities for the application of machine learning and artificial intelligence (AI) to help answer key questions in the field. We provide an overview of the state of the science, highlighting current challenges and opportunities for utilisation of AI approaches to move the field forward in the areas of genetics, experimental medicine, drug discovery and trials optimisation, imaging, and prevention. Machine learning methods can enhance results of genetic studies, help determine biological effects and facilitate the identification of drug targets based on genetic and transcriptomic information. The use of unsupervised learning for understanding disease mechanisms for drug discovery is promising, while analysis of multimodal data sets to characterise and quantify disease severity and subtype are also beginning to contribute to optimisation of clinical trial recruitment. Data-driven experimental medicine is needed to analyse data across modalities and develop novel algorithms to translate insights from animal models to human disease biology. AI methods in neuroimaging outperform traditional approaches for diagnostic classification, and although challenges around validation and translation remain, there is optimism for their meaningful integration to clinical practice in the near future. AI-based models can also clarify our understanding of the causality and commonality of dementia risk factors, informing and improving risk prediction models along with the development of preventative interventions. The complexity and heterogeneity of dementia requires an alternative approach beyond traditional design and analytical approaches. Although not yet widely used in dementia research, machine learning and AI have the potential to unlock current challenges and advance precision dementia medicine.

Differentiation of Subjective Cognitive Decline, Mild Cognitive Impairment, and Dementia Using qEEG/ERP-Based Cognitive Testing and Volumetric MRI in an Outpatient Specialty Memory Clinic

BACKGROUND

Distinguishing between subjective cognitive decline (SCD), mild cognitive impairment (MCI), and dementia in a scalable, accessible way is important to promote earlier detection and intervention.

OBJECTIVE

We investigated diagnostic categorization using an FDA-cleared quantitative electroencephalographic/event-related potential (qEEG/ERP)-based cognitive testing system (eVox® by Evoke Neuroscience) combined with an automated volumetric magnetic resonance imaging (vMRI) tool (Neuroreader® by Brainreader).

METHODS

Patients who self-presented with memory complaints were assigned to a diagnostic category by dementia specialists based on clinical history, neurologic exam, neuropsychological testing, and laboratory results. In addition, qEEG/ERP (n = 161) and quantitative vMRI (n = 111) data were obtained. A multinomial logistic regression model was used to determine significant predictors of cognitive diagnostic category (SCD, MCI, or dementia) using all available qEEG/ERP features and MRI volumes as the independent variables and controlling for demographic variables. Area under the Receiver Operating Characteristic curve (AUC) was used to evaluate the diagnostic accuracy of the prediction models.

RESULTS

The qEEG/ERP measures of Reaction Time, Commission Errors, and P300b Amplitude were significant predictors (AUC = 0.79) of cognitive category. Diagnostic accuracy increased when volumetric MRI measures, specifically left temporal lobe volume, were added to the model (AUC = 0.87).

CONCLUSION

This study demonstrates the potential of a primarily physiological diagnostic model for differentiating SCD, MCI, and dementia using qEEG/ERP-based cognitive testing, especially when combined with volumetric brain MRI. The accessibility of qEEG/ERP and vMRI means that these tools can be used as adjuncts to clinical assessments to help increase the diagnostic certainty of SCD, MCI, and dementia.

Impact of normative brain volume reports on the diagnosis of neurodegenerative dementia disorders in neuroradiology: A real-world, clinical practice study

Background: Normative brain volume reports (NBVR) are becoming more available in the work-up of patients with suspected dementia disorders, potentially leveraging the value of structural MRI in clinical settings. The present study aims to investigate the impact of NBVRs on the diagnosis of neurodegenerative dementia disorders in real-world clinical practice.

Methods: We retrospectively analyzed data of 112 memory clinic patients, who were consecutively referred for MRI and 18F-fluorodeoxyglucose (FDG) positron emission tomography (PET) during a 12-month period. Structural MRI was assessed by two residents with 2 and 3 years of neuroimaging experience. Statements and diagnostic confidence regarding the presence of a neurodegenerative disorder in general (first level) and Alzheimer’s disease (AD) pattern in particular (second level) were recorded without and with NBVR information. FDG-PET served as the reference standard.

Results: Overall, despite a trend towards increased accuracy, the impact of NBVRs on diagnostic accuracy was low and non-significant. We found a significant drop of sensitivity (0.75–0.58; p < 0.001) and increase of specificity (0.62–0.85; p < 0.001) for rater 1 at identifying patients with neurodegenerative dementia disorders. Diagnostic confidence increased for rater 2 (p < 0.001).

Conclusions: Overall, NBVRs had a limited impact on diagnostic accuracy in real-world clinical practice. Potentially, NBVR might increase diagnostic specificity and confidence of neuroradiology residents. To this end, a well-defined framework for integration of NBVR in the diagnostic process and improved algorithms of NBVR generation are essential.

Volumetric Assessment of Hippocampus and Subcortical Gray Matter Regions in Alzheimer Disease and Amnestic Mild Cognitive Impairment

BACKGROUND

Quantitative MRI assessment methods have limited utility due to a lack of standardized methods and measures for Alzheimer disease (AD) and amnestic mild cognitive impairment (aMCI).

OBJECTIVE

To employ a relatively new and easy-to-use quantitative assessment method to reveal volumetric changes in subcortical gray matter (GM) regions, hippocampus, and global intracranial structures as well as the diagnostic performance and best thresholds of total hippocampal volumetry in individuals with AD and those with aMCI.

METHOD

A total of 74 individuals-37 with mild to moderate AD, 19 with aMCI, and 18 with normal cognition (NC)-underwent a 3T MRI. Fully automated segmentation and volumetric measurements were performed.

RESULTS

The AD and aMCI groups had smaller volumes of amygdala, nucleus accumbens, and hippocampus compared with the NC group. These same two groups had significantly smaller total white matter volume than the NC group. The AD group had smaller total GM volume compared with the aMCI and NC groups. The thalamus in the AD group showed a subtle atrophy. There were no significant volumetric differences in the caudate nucleus, putamen, or globus pallidus between the groups.

CONCLUSION

The amygdala and nucleus accumbens showed atrophy comparable to the hippocampal atrophy in both the AD and aMCI groups, which may contribute to cognitive impairment. Hippocampal volumetry is a reliable tool for differentiating between AD and NC groups but has substantially less power in differentiating between AD and aMCI groups. The loss of total GM volume differentiates AD from aMCI and NC.

Technical and clinical validation of commercial automated volumetric MRI tools for dementia diagnosis-a systematic review

Developments in neuroradiological MRI analysis offer promise in enhancing objectivity and consistency in dementia diagnosis through the use of quantitative volumetric reporting tools (QReports). Translation into clinical settings should follow a structured framework of development, including technical and clinical validation steps. However, published technical and clinical validation of the available commercial/proprietary tools is not always easy to find and pathways for successful integration into the clinical workflow are varied. The quantitative neuroradiology initiative (QNI) framework highlights six necessary steps for the development, validation and integration of quantitative tools in the clinic. In this paper, we reviewed the published evidence regarding regulatory-approved QReports for use in the memory clinic and to what extent this evidence fulfils the steps of the QNI framework. We summarize unbiased technical details of available products in order to increase the transparency of evidence and present the range of reporting tools on the market. Our intention is to assist neuroradiologists in making informed decisions regarding the adoption of these methods in the clinic. For the 17 products identified, 11 companies have published some form of technical validation on their methods, but only 4 have published clinical validation of their QReports in a dementia population. Upon systematically reviewing the published evidence for regulatory-approved QReports in dementia, we concluded that there is a significant evidence gap in the literature regarding clinical validation, workflow integration and in-use evaluation of these tools in dementia MRI diagnosis.

Automatic quality control of brain T1-weighted magnetic resonance images for a clinical data warehouse

Many studies on machine learning (ML) for computer-aided diagnosis have so far been mostly restricted to high-quality research data. Clinical data warehouses, gathering routine examinations from hospitals, offer great promises for training and validation of ML models in a realistic setting. However, the use of such clinical data warehouses requires quality control (QC) tools. Visual QC by experts is time-consuming and does not scale to large datasets. In this paper, we propose a convolutional neural network (CNN) for the automatic QC of 3D T1-weighted brain MRI for a large heterogeneous clinical data warehouse. To that purpose, we used the data warehouse of the hospitals of the Greater Paris area (Assistance Publique-Hôpitaux de Paris [AP-HP]). Specifically, the objectives were: 1) to identify images which are not proper T1-weighted brain MRIs; 2) to identify acquisitions for which gadolinium was injected; 3) to rate the overall image quality. We used 5000 images for training and validation and a separate set of 500 images for testing. In order to train/validate the CNN, the data were annotated by two trained raters according to a visual QC protocol that we specifically designed for application in the setting of a data warehouse. For objectives 1 and 2, our approach achieved excellent accuracy (balanced accuracy and F1-score >90%), similar to the human raters. For objective 3, the performance was good but substantially lower than that of human raters. Nevertheless, the automatic approach accurately identified (balanced accuracy and F1-score >80%) low quality images, which would typically need to be excluded. Overall, our approach shall be useful for exploiting hospital data warehouses in medical image computing.

The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation

Evaluating binary classifications is a pivotal task in statistics and machine learning, because it can influence decisions in multiple areas, including for example prognosis or therapies of patients in critical conditions. The scientific community has not agreed on a general-purpose statistical indicator for evaluating two-class confusion matrices (having true positives, true negatives, false positives, and false negatives) yet, even if advantages of the Matthews correlation coefficient (MCC) over accuracy and F1 score have already been shown.In this manuscript, we reaffirm that MCC is a robust metric that summarizes the classifier performance in a single value, if positive and negative cases are of equal importance. We compare MCC to other metrics which value positive and negative cases equally: balanced accuracy (BA), bookmaker informedness (BM), and markedness (MK). We explain the mathematical relationships between MCC and these indicators, then show some use cases and a bioinformatics scenario where these metrics disagree and where MCC generates a more informative response.Additionally, we describe three exceptions where BM can be more appropriate: analyzing classifications where dataset prevalence is unrepresentative, comparing classifiers on different datasets, and assessing the random guessing level of a classifier. Except in these cases, we believe that MCC is the most informative among the single metrics discussed, and suggest it as standard measure for scientists of all fields. A Matthews correlation coefficient close to +1, in fact, means having high values for all the other confusion matrix metrics. The same cannot be said for balanced accuracy, markedness, bookmaker informedness, accuracy and F1 score.

Radiological classification of dementia from anatomical MRI assisted by machine learning-derived maps

BACKGROUND AND PURPOSE

Many artificial intelligence tools are currently being developed to assist diagnosis of dementia from magnetic resonance imaging (MRI). However, these tools have so far been difficult to integrate in the clinical routine workflow. In this work, we propose a new simple way to use them and assess their utility for improving diagnostic accuracy.

MATERIALS AND METHODS

We studied 34 patients with early-onset Alzheimer's disease (EOAD), 49 with late-onset AD (LOAD), 39 with frontotemporal dementia (FTD) and 24 with depression from the pre-existing cohort CLIN-AD. Support vector machine (SVM) automatic classifiers using 3D T1 MRI were trained to distinguish: LOAD vs. Depression, FTD vs. LOAD, EOAD vs. Depression, EOAD vs. FTD. We extracted SVM weight maps, which are tridimensional representations of discriminant atrophy patterns used by the classifier to take its decisions and we printed posters of these maps. Four radiologists (2 senior neuroradiologists and 2 unspecialized junior radiologists) performed a visual classification of the 4 diagnostic pairs using 3D T1 MRI. Classifications were performed twice: first with standard radiological reading and then using SVM weight maps as a guide.

RESULTS

Diagnostic performance was significantly improved by the use of the weight maps for the two junior radiologists in the case of FTD vs. EOAD. Improvement was over 10 points of diagnostic accuracy.

CONCLUSION

This tool can improve the diagnostic accuracy of junior radiologists and could be integrated in the clinical routine workflow.