Clinically Available Software for Automatic Brain Volumetry: Comparisons of Volume Measurements and Validation of Intermethod Reliability

Comparison of Inter-Method Agreement and Reliability for Automatic Brain Volumetry Using Three Different Clinically Available Software Packages

Background and Objectives: No comparative study has evaluated the inter-method agreement and reliability between Heuron AD and other clinically available brain volumetric software packages. Hence, we aimed to investigate the inter-method agreement and reliability of three clinically available brain volumetric software packages: FreeSurfer (FS), NeuroQuant® (NQ), and Heuron AD (HAD). Materials and Methods: In this study, we retrospectively included 78 patients who underwent conventional three-dimensional (3D) T1-weighed imaging (T1WI) to evaluate their memory impairment, including 21 with normal objective cognitive function, 24 with mild cognitive impairment, and 33 with Alzheimer's disease (AD). All 3D T1WI scans were analyzed using three different volumetric software packages. Repeated-measures analysis of variance, intraclass correlation coefficient, effect size measurements, and Bland-Altman analysis were used to evaluate the inter-method agreement and reliability. Results: The measured volumes demonstrated substantial to almost perfect agreement for most brain regions bilaterally, except for the bilateral globi pallidi. However, the volumes measured using the three software packages showed significant mean differences for most brain regions, with consistent systematic biases and wide limits of agreement in the Bland-Altman analyses. The pallidum showed the largest effect size in the comparisons between NQ and FS (5.20-6.93) and between NQ and HAD (2.01-6.17), while the cortical gray matter showed the largest effect size in the comparisons between FS and HAD (0.79-1.91). These differences and variations between the software packages were also observed in the subset analyses of 45 patients without AD and 33 patients with AD. Conclusions: Despite their favorable reliability, the software-based brain volume measurements showed significant differences and systematic biases in most regions. Thus, these volumetric measurements should be interpreted based on the type of volumetric software used, particularly for smaller structures. Moreover, users should consider the replaceability-related limitations when using these packages in real-world practice.

Contrast-enhanced to non-contrast-enhanced image translation to exploit a clinical data warehouse of T1-weighted brain MRI

BACKGROUND

Clinical data warehouses provide access to massive amounts of medical images, but these images are often heterogeneous. They can for instance include images acquired both with or without the injection of a gadolinium-based contrast agent. Harmonizing such data sets is thus fundamental to guarantee unbiased results, for example when performing differential diagnosis. Furthermore, classical neuroimaging software tools for feature extraction are typically applied only to images without gadolinium. The objective of this work is to evaluate how image translation can be useful to exploit a highly heterogeneous data set containing both contrast-enhanced and non-contrast-enhanced images from a clinical data warehouse.

METHODS

We propose and compare different 3D U-Net and conditional GAN models to convert contrast-enhanced T1-weighted (T1ce) into non-contrast-enhanced (T1nce) brain MRI. These models were trained using 230 image pairs and tested on 77 image pairs from the clinical data warehouse of the Greater Paris area.

RESULTS

Validation using standard image similarity measures demonstrated that the similarity between real and synthetic T1nce images was higher than between real T1nce and T1ce images for all the models compared. The best performing models were further validated on a segmentation task. We showed that tissue volumes extracted from synthetic T1nce images were closer to those of real T1nce images than volumes extracted from T1ce images.

CONCLUSION

We showed that deep learning models initially developed with research quality data could synthesize T1nce from T1ce images of clinical quality and that reliable features could be extracted from the synthetic images, thus demonstrating the ability of such methods to help exploit a data set coming from a clinical data warehouse.

Magnetic resonance image-based brain age as a discriminator of dementia conversion in patients with amyloid-negative amnestic mild cognitive impairment

Patients with amyloid-negative amnestic mild cognitive impairment (MCI) have a conversion rate of approximately 10% to dementia within 2 years. We aimed to investigate whether brain age is an important factor in predicting conversion to dementia in patients with amyloid-negative amnestic MCI. We conducted a retrospective cohort study of patients with amyloid-negative amnestic MCI. All participants underwent detailed neuropsychological evaluation, brain magnetic resonance imaging (MRI), and [18F]-florbetaben positron emission tomography. Brain age was determined by the volumetric assessment of 12 distinct brain regions using an automatic segmentation software. During the follow-up period, 38% of the patients converted from amnestic MCI to dementia. Further, 73% of patients had a brain age greater than their actual chronological age. When defining 'survival' as the non-conversion of MCI to dementia, these groups differed significantly in survival probability (p = 0.036). The low-educated female group with a brain age greater than their actual age had the lowest survival rate among all groups. Our findings suggest that the MRI-based brain age used in this study can contribute to predicting conversion to dementia in patients with amyloid-negative amnestic MCI.

Reliability of automated brain volumetric analysis: A test by comparing NeuroQuant and volBrain software

BACKGROUND AND PURPOSE

Brain volume analysis from magnetic resonance imaging (MRI) is gaining an important role in neurological diagnosis. This study compares the volumes of brain segments measured by two automated brain analysis software, NeuroQuant (NQ), and volBrain (VB) in order to test their reliability in brain volumetry.

METHODS

Using NQ and VB software, the same brain segment volumes were calculated and compared, taken from 56 patients scanned under the same MRI sequence. These segments were intracranial cavity, putamen, thalamus, amygdala, whole brain, cerebellum, white matter, and hippocampus. The paired t-test method has been used to determine if there was a significant difference in these measurements. The interclass correlation (ICC) is used to test inter-method reliability between the two software. Finally, regression analysis was used to examine the possibility of linear correlation.

RESULTS

In all brain segments tested but hippocampus, significant differences were found. ICC presents satisfactory to excellent reliability in all brain segments except thalamus and amygdala for which reliability has been proven to be poor. In most cases, linear correlation was found.

CONCLUSIONS

The significant differences found in the majority of the tested brain segments are raising questions about the reliability of automated brain analysis as a quantitative tool. Strong linear correlation of the volumetric measurements and good reliability indicates that, each software provides good qualitative information of brain structures size.

Development and validation of a deep learning-based automatic segmentation model for assessing intracranial volume: comparison with NeuroQuant, FreeSurfer, and SynthSeg

Background and purpose

To develop and validate a deep learning-based automatic segmentation model for assessing intracranial volume (ICV) and to compare the accuracy determined by NeuroQuant (NQ), FreeSurfer (FS), and SynthSeg.

Materials and methods

This retrospective study included 60 subjects [30 Alzheimer's disease (AD), 21 mild cognitive impairment (MCI), 9 cognitively normal (CN)] from a single tertiary hospital for the training and validation group (50:10). The test group included 40 subjects (20 AD, 10 MCI, 10 CN) from the ADNI dataset. We propose a robust ICV segmentation model based on the foundational 2D UNet architecture trained with four types of input images (both single and multimodality using scaled or unscaled T1-weighted and T2-FLAIR MR images). To compare with our model, NQ, FS, and SynthSeg were also utilized in the test group. We evaluated the model performance by measuring the Dice similarity coefficient (DSC) and average volume difference.

Results

The single-modality model trained with scaled T1-weighted images showed excellent performance with a DSC of 0.989 ± 0.002 and an average volume difference of 0.46% ± 0.38%. Our multimodality model trained with both unscaled T1-weighted and T2-FLAIR images showed similar performance with a DSC of 0.988 ± 0.002 and an average volume difference of 0.47% ± 0.35%. The overall average volume difference with our model showed relatively higher accuracy than NQ (2.15% ± 1.72%), FS (3.69% ± 2.93%), and SynthSeg (1.88% ± 1.18%). Furthermore, our model outperformed the three others in each subgroup of patients with AD, MCI, and CN subjects.

Conclusion

Our deep learning-based automatic ICV segmentation model showed excellent performance for the automatic evaluation of ICV.

Comparison of Normative Percentiles of Brain Volume Obtained from NeuroQuant® vs. DeepBrain® in the Korean Population: Correlation with Cranial Shape

Purpose

This study aimed to compare the volume and normative percentiles of brain volumetry in the Korean population using quantitative brain volumetric MRI analysis tools NeuroQuant® (NQ) and DeepBrain® (DB), and to evaluate whether the differences in the normative percentiles of brain volumetry between the two tools is related to cranial shape.

Materials and Methods

In this retrospective study, we analyzed the brain volume reports obtained from NQ and DB in 163 participants without gross structural brain abnormalities. We measured three-dimensional diameters to evaluate the cranial shape on T1-weighted images. Statistical analyses were performed using intra-class correlation coefficients and linear correlations.

Results

The mean normative percentiles of the thalamus (90.8 vs. 63.3 percentile), putamen (90.0 vs. 60.0 percentile), and parietal lobe (80.1 vs. 74.1 percentile) were larger in the NQ group than in the DB group, whereas that of the occipital lobe (18.4 vs. 68.5 percentile) was smaller in the NQ group than in the DB group. We found a significant correlation between the mean normative percentiles obtained from the NQ and cranial shape: the mean normative percentile of the occipital lobe increased with the anteroposterior diameter and decreased with the craniocaudal diameter.

Conclusion

The mean normative percentiles obtained from NQ and DB differed significantly for many brain regions, and these differences may be related to cranial shape.

FDA-approved machine learning algorithms in neuroradiology: A systematic review of the current evidence for approval

Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as "medical devices" to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.

Brain Parcellation Repeatability and Reproducibility Using Conventional and Quantitative 3D MR Imaging

BACKGROUND AND PURPOSE

Automatic brain parcellation is typically performed on dedicated MR imaging sequences, which require valuable examination time. In this study, a 3D MR imaging quantification sequence to retrieve R1 and R2 relaxation rates and proton density maps was used to synthesize a T1-weighted image stack for brain volume measurement, thereby combining image data for multiple purposes. The repeatability and reproducibility of using the conventional and synthetic input data were evaluated.

MATERIALS AND METHODS

Twelve subjects with a mean age of 54 years were scanned twice at 1.5T and 3T with 3D-QALAS and a conventionally acquired T1-weighted sequence. Using SyMRI, we converted the R1, R2, and proton density maps into synthetic T1-weighted images. Both the conventional T1-weighted and the synthetic 3D-T1-weighted inversion recovery images were processed for brain parcellation by NeuroQuant. Bland-Altman statistics were used to correlate the volumes of 12 brain structures. The coefficient of variation was used to evaluate the repeatability.

RESULTS

A high correlation with medians of 0.97 for 1.5T and 0.92 for 3T was found. A high repeatability was shown with a median coefficient of variation of 1.2% for both T1-weighted and synthetic 3D-T1-weighted inversion recovery at 1.5T, and 1.5% for T1-weighted imaging and 4.4% for synthetic 3D-T1-weighted inversion recovery at 3T. However, significant biases were observed between the methods and field strengths.

CONCLUSIONS

It is possible to perform MR imaging quantification of R1, R2, and proton density maps to synthesize a 3D-T1-weighted image stack, which can be used for automatic brain parcellation. Synthetic parameter settings should be reinvestigated to reduce the observed bias.

Development and validation of deep learning-based automatic brain segmentation for East Asians: A comparison with Freesurfer

Purpose

To develop and validate deep learning-based automatic brain segmentation for East Asians with comparison to data for healthy controls from Freesurfer based on a ground truth.

Methods

A total of 30 healthy participants were enrolled and underwent T1-weighted magnetic resonance imaging (MRI) using a 3-tesla MRI system. Our Neuro I software was developed based on a three-dimensional convolutional neural networks (CNNs)-based, deep-learning algorithm, which was trained using data for 776 healthy Koreans with normal cognition. Dice coefficient (D) was calculated for each brain segment and compared with control data by paired t-test. The inter-method reliability was assessed by intraclass correlation coefficient (ICC) and effect size. Pearson correlation analysis was applied to assess the relationship between D values for each method and participant ages.

Results

The D values obtained from Freesurfer (ver6.0) were significantly lower than those from Neuro I. The histogram of the Freesurfer results showed remarkable differences in the distribution of D values from Neuro I. Overall, D values obtained by Freesurfer and Neuro I showed positive correlations, but the slopes and intercepts were significantly different. It was showed the largest effect sizes ranged 1.07-3.22, and ICC also showed significantly poor to moderate correlations between the two methods (0.498 ≤ ICC ≤ 0.688). For Neuro I, D values resulted in reduced residuals when fitting data to a line of best fit, and indicated consistent values corresponding to each age, even in young and older adults.

Conclusion

Freesurfer and Neuro I were not equivalent when compared to a ground truth, where Neuro I exhibited higher performance. We suggest that Neuro I is a useful alternative for the assessment of the brain volume.

Agreement and Reliability between Clinically Available Software Programs in Measuring Volumes and Normative Percentiles of Segmented Brain Regions

OBJECTIVE

To investigate the agreement and reliability of estimating the volumes and normative percentiles (N%) of segmented brain regions among NeuroQuant (NQ), DeepBrain (DB), and FreeSurfer (FS) software programs, focusing on the comparison between NQ and DB.

MATERIALS AND METHODS

Three-dimensional T1-weighted images of 145 participants (48 healthy participants, 50 patients with mild cognitive impairment, and 47 patients with Alzheimer's disease) from a single medical center (SMC) dataset and 130 participants from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset were included in this retrospective study. All images were analyzed with DB, NQ, and FS software to obtain volume estimates and N% of various segmented brain regions. We used Bland-Altman analysis, repeated measures ANOVA, reproducibility coefficient, effect size, and intraclass correlation coefficient (ICC) to evaluate inter-method agreement and reliability.

RESULTS

Among the three software programs, the Bland-Altman plot showed a substantial bias, the ICC showed a broad range of reliability (0.004-0.97), and repeated-measures ANOVA revealed significant mean volume differences in all brain regions. Similarly, the volume differences of the three software programs had large effect sizes in most regions (0.73-5.51). The effect size was largest in the pallidum in both datasets and smallest in the thalamus and cerebral white matter in the SMC and ADNI datasets, respectively. N% of NQ and DB showed an unacceptably broad Bland-Altman limit of agreement in all brain regions and a very wide range of ICC values (-0.142-0.844) in most brain regions.

CONCLUSION

NQ and DB showed significant differences in the measured volume and N%, with limited agreement and reliability for most brain regions. Therefore, users should be aware of the lack of interchangeability between these software programs when they are applied in clinical practice.

[Brain MRI-Based Artificial Intelligence Software in Patients with Neurodegenerative Diseases: Current Status]

The incidence of neurodegenerative diseases in the older population has increased in recent years. A considerable number of studies have been performed to characterize these diseases. Imaging analysis is an important biomarker for the diagnosis of neurodegenerative disease. Objective and reliable assessment and precise detection are important for the early diagnosis of neurodegenerative diseases. Artificial intelligence (AI) using brain MRI applied to the study of neurodegenerative diseases could promote early diagnosis and optimal decisions for treatment plans. MRI-based AI software have been developed and studied worldwide. Representatively, there are MRI-based volumetry and segmentation software. In this review, we present the development process of brain volumetry analysis software in neurodegenerative diseases, currently used and developed AI software for neurodegenerative disease in the Republic of Korea, probable uses of AI in the future, and AI software limitations.

Updated Review of the Evidence Supporting the Medical and Legal Use of NeuroQuant® and NeuroGage® in Patients With Traumatic Brain Injury

Over 40 years of research have shown that traumatic brain injury affects brain volume. However, technical and practical limitations made it difficult to detect brain volume abnormalities in patients suffering from chronic effects of mild or moderate traumatic brain injury. This situation improved in 2006 with the FDA clearance of NeuroQuant®, a commercially available, computer-automated software program for measuring MRI brain volume in human subjects. More recent strides were made with the introduction of NeuroGage®, commercially available software that is based on NeuroQuant® and extends its utility in several ways. Studies using these and similar methods have found that most patients with chronic mild or moderate traumatic brain injury have brain volume abnormalities, and several of these studies found-surprisingly-more abnormal enlargement than atrophy. More generally, 102 peer-reviewed studies have supported the reliability and validity of NeuroQuant® and NeuroGage®. Furthermore, this updated version of a previous review addresses whether NeuroQuant® and NeuroGage® meet the Daubert standard for admissibility in court. It concludes that NeuroQuant® and NeuroGage® meet the Daubert standard based on their reliability, validity, and objectivity. Due to the improvements in technology over the years, these brain volumetric techniques are practical and readily available for clinical or forensic use, and thus they are important tools for detecting signs of brain injury.

Technical and clinical validation of commercial automated volumetric MRI tools for dementia diagnosis-a systematic review

Developments in neuroradiological MRI analysis offer promise in enhancing objectivity and consistency in dementia diagnosis through the use of quantitative volumetric reporting tools (QReports). Translation into clinical settings should follow a structured framework of development, including technical and clinical validation steps. However, published technical and clinical validation of the available commercial/proprietary tools is not always easy to find and pathways for successful integration into the clinical workflow are varied. The quantitative neuroradiology initiative (QNI) framework highlights six necessary steps for the development, validation and integration of quantitative tools in the clinic. In this paper, we reviewed the published evidence regarding regulatory-approved QReports for use in the memory clinic and to what extent this evidence fulfils the steps of the QNI framework. We summarize unbiased technical details of available products in order to increase the transparency of evidence and present the range of reporting tools on the market. Our intention is to assist neuroradiologists in making informed decisions regarding the adoption of these methods in the clinic. For the 17 products identified, 11 companies have published some form of technical validation on their methods, but only 4 have published clinical validation of their QReports in a dementia population. Upon systematically reviewing the published evidence for regulatory-approved QReports in dementia, we concluded that there is a significant evidence gap in the literature regarding clinical validation, workflow integration and in-use evaluation of these tools in dementia MRI diagnosis.

Evaluation of Reproducibility of Brain Volumetry between Commercial Software, Inbrain and Established Research Purpose Method, FreeSurfer

Background and Purpose

We aimed to determine the intermethod reproducibility between the commercial software Inbrain (MIDAS IT) and the established research-purpose method FreeSurfer, as well as the effect of MRI resolution and the pathological condition of subjects on their intermethod reproducibility.

Methods

This study included 45 healthy volunteers and 85 patients with mild cognitive impairment (MCI). In 43 of the 85 patients with MCI, three-dimensional, T1-weighted MRI data were obtained at an in-plane resolution of 1.2 mm. The data of the remaining 42 patients with MCI and the healthy volunteers were obtained at an in-plane resolution of 1.0 mm. The within-subject coefficient of variation (CoV), intraclass correlation coefficient (ICC), and effect size were calculated, and means were compared using paired t-tests. The parameters obtained at 1.0-mm and 1.2-mm resolutions in patients with MCI were compared to evaluate the effect of the in-plane resolution on the intermethod reproducibility. The parameters obtained at a 1.0-mm in-plane resolution in patients with MCI and healthy volunteers were used to analyze the effect of subject condition on intermethod reproducibility.

Results

Overall the two methods showed excellent reproducibility across all regions of the brain (CoV=0.5–3.9, ICC=0.93 to >0.99). In the subgroup of healthy volunteers, the intermethod reliability was only good in some regions (frontal, temporal, cingulate, and insular). The intermethod reproducibility was better in the 1.0-mm group than the 1.2-mm group in all regions other than the nucleus accumbens.

Conclusions

Inbrain and FreeSurfer showed good-to-excellent intermethod reproducibility for volumetric measurements. Nevertheless, some noticeable differences were found based on subject condition, image resolution, and brain region.

Expert Opinions and Recommendations for the Clinical Use of Quantitative Analysis Software for MRI-Based Brain Volumetry

치매를 비롯한 퇴행성 신경 질환의 초기 진단에 자기공명영상을 이용한 뇌 위축 평가와 정량적 용적 분석이 중요하다. 뇌 위축의 시각적 평가는 주관적으로 평가자에 따라 다른 결과를 보여주기 때문에, 객관적인 결과를 제공하면서 임상 적용도 가능한 소프트웨어의 수요와 개발이 늘어나고 있다. 이러한 임상용 소프트웨어의 실제 임상 적용은 영상 검사의 표준화가 선행되어야 하고, 개발된 소프트웨어의 검증이 반드시 필요하다. 따라서 대한신경두경부영상의학회는 뇌용적 분석 임상용 소프트웨어의 임상적 활용에 대한 의견을 제시하기 위해 전문위원회를 구성하고 현재까지 발표된 연구를 정리하였다. 그리고, 정량화 분석을 위한 영상 검사의 표준화 및 소프트웨어의 임상 적용에 대한 전문가 의견을 제시하기 위하여 공동 작업을 수행하였다. 본 종설에서는 뇌 자기공명영상의 정량화 분석의 필요성 및 배경, 정량화 분석을 위한 임상용 소프트웨어의 소개 및 기존의 표준품(reference standard)과의 진단능 비교, 영상 획득의 표준화, 분석 및 평가의 표준화, 소프트웨어의 임상 적용에 대한 전문가 의견, 제한점 및 대처 방법 등 대한신경두경부영상의학회의 전문가 권고안을 소개하는 것이 목적이다.