The effect of initial teaching on evaluation of left ventricular volumes by cardiovascular magnetic resonance imaging: comparison between complete and intermediate beginners and experienced observers

Multivendor Evaluation of Automated MRI Postprocessing of Biventricular Size and Function for Children With and Without Congenital Heart Defects

BACKGROUND

Manually segmenting cardiac structures is time-consuming and produces variability in MRI assessments. Automated segmentation could solve this. However, current software is developed for adults without congenital heart defects (CHD).

PURPOSE

To evaluate automated segmentation of left ventricle (LV) and right ventricle (RV) for pediatric MRI studies.

STUDY TYPE

Retrospective comparative study.

POPULATION

Twenty children per group of: healthy children, LV-CHD, tetralogy of Fallot (ToF), and univentricular CHD, aged 11.7 [8.9-16.0], 14.2 [10.6-15.7], 14.6 [11.6-16.4], and 12.2 [10.2-14.9] years, respectively.

SEQUENCE/FIELD STRENGTH

Balanced steady-state free precession at 1.5 T.

ASSESSMENT

Biventricular volumes and masses were calculated from a short-axis stack of images, which were segmented manually and using two fully automated software suites (Medis Suite 3.2, Medis, Leiden, the Netherlands and SuiteHeart 5.0, Neosoft LLC, Pewaukee, USA). Fully automated segmentations were manually adjusted to provide two further sets of segmentations. Fully automated and adjusted automated segmentation were compared to manual segmentation. Segmentation times and reproducibility for each method were assessed.

STATISTICAL TESTS

Bland Altman analysis and intraclass correlation coefficients (ICC) were used to compare volumes and masses between methods. Postprocessing times were compared by paired t-tests.

RESULTS

Fully automated methods provided good segmentation (ICC > 0.90 compared to manual segmentation) for the LV in the healthy and left-sided CHD groups (eg LV-EDV difference for healthy children 1.4 ± 11.5 mL, ICC: 0.97, for Medis and 3.0 ± 12.2 mL, ICC: 0.96 for SuiteHeart). Both automated methods gave larger errors (ICC: 0.62-0.94) for the RV in these populations, and for all structures in the ToF and univentricular CHD groups. Adjusted automated segmentation agreed well with manual segmentation (ICC: 0.71-1.00), improved reproducibility and reduced segmentation time in all patient groups, compared to manual segmentation.

DATA CONCLUSION

Fully automated segmentation eliminates observer variability but may produce large errors compared to manual segmentation. Manual adjustments reduce these errors, improve reproducibility, and reduce postprocessing times compared to manual segmentation. Adjusted automated segmentation is reasonable in children with and without CHD.

EVIDENCE LEVEL

3.

TECHNICAL EFFICACY

Stage 2.

Lazy Luna: Extendible software for multilevel reader comparison in cardiovascular magnetic resonance imaging

BACKGROUND AND OBJECTIVES

Cardiovascular Magnetic Resonance (CMR) imaging is a growing field with increasing diagnostic utility in clinical routine. Quantitative diagnostic parameters are typically calculated based on contours or points provided by readers, e.g. natural intelligences (NI) such as clinicians or researchers, and artificial intelligences (AI). As clinical applications multiply, evaluating the precision and reproducibility of quantitative parameters becomes increasingly important. Although segmentation challenges for AIs and guidelines for clinicians provide quality assessments and regulation, the methods ought to be combined and streamlined for clinical applications. The goal of the developed software, Lazy Luna (LL), is to offer a flexible evaluation tool that is readily extendible to new sequences and scientific endeavours.

METHODS

An interface was designed for LL, which allows for comparing annotated CMR images. Geometric objects ensure precise calculations of metric values and clinical results regardless of whether annotations originate from AIs or NIs. A graphical user interface (GUI) is provided to make the software available to non-programmers. The GUI allows for an interactive inspection of image datasets as well as implementing tracing procedures, which follow statistical reader differences in clinical results to their origins in individual image contours. The backend software builds on a set of meta-classes, which can be extended to new imaging sequences and clinical parameters. Following an agile development procedure with clinical feedback allows for a quick implementation of new classes, figures and tables for evaluation.

RESULTS

Two application cases present LL's extendibility to clinical evaluation and AI development contexts. The first concerns T1 parametric mapping images segmented by two expert readers. Quantitative result differences are traced to reveal typical segmentation dissimilarities from which these differences originate. The meta-classes are extended to this new application scenario. The second applies to the open source Late Gadolinium Enhancement (LGE) quantification challenge for AI developers "Emidec", which illustrates LL's usability as open source software.

CONCLUSION

The presented software Lazy Luna allows for an automated multilevel comparison of readers as well as identifying qualitative reasons for statistical reader differences. The open source software LL can be extended to new application cases in the future.

Introduction of Lazy Luna an automatic software-driven multilevel comparison of ventricular function quantification in cardiovascular magnetic resonance imaging

Cardiovascular magnetic resonance imaging is the gold standard for cardiac function assessment. Quantification of clinical results (CR) requires precise segmentation. Clinicians statistically compare CRs to ensure reproducibility. Convolutional Neural Network developers compare their results via metrics. Aim: Introducing software capable of automatic multilevel comparison. A multilevel analysis covering segmentations and CRs builds on a generic software backend. Metrics and CRs are calculated with geometric accuracy. Segmentations and CRs are connected to track errors and their effects. An interactive GUI makes the software accessible to different users. The software’s multilevel comparison was tested on a use case based on cardiac function assessment. The software shows good reader agreement in CRs and segmentation metrics (Dice > 90%). Decomposing differences by cardiac position revealed excellent agreement in midventricular slices: > 90% but poorer segmentations in apical (> 71%) and basal slices (> 74%). Further decomposition by contour type locates the largest millilitre differences in the basal right cavity (> 3 ml). Visual inspection shows these differences being caused by different basal slice choices. The software illuminated reader differences on several levels. Producing spreadsheets and figures concerning metric values and CR differences was automated. A multilevel reader comparison is feasible and extendable to other cardiac structures in the future.

Balancing Speed and Accuracy in Cardiac Magnetic Resonance Function Post-Processing: Comparing 2 Levels of Automation in 3 Vendors to Manual Assessment

Automating cardiac function assessment on cardiac magnetic resonance short-axis cines is faster and more reproducible than manual contour-tracing; however, accurately tracing basal contours remains challenging. Three automated post-processing software packages (Level 1) were compared to manual assessment. Subsequently, automated basal tracings were manually adjusted using a standardized protocol combined with software package-specific relative-to-manual standard error correction (Level 2). All post-processing was performed in 65 healthy subjects. Manual contour-tracing was performed separately from Level 1 and 2 automated analysis. Automated measurements were considered accurate when the difference was equal or less than the maximum manual inter-observer disagreement percentage. Level 1 (2.1 ± 1.0 min) and Level 2 automated (5.2 ± 1.3 min) were faster and more reproducible than manual (21.1 ± 2.9 min) post-processing, the maximum inter-observer disagreement was 6%. Compared to manual, Level 1 automation had wide limits of agreement. The most reliable software package obtained more accurate measurements in Level 2 compared to Level 1 automation: left ventricular end-diastolic volume, 98% and 53%; ejection fraction, 98% and 60%; mass, 70% and 3%; right ventricular end-diastolic volume, 98% and 28%; ejection fraction, 80% and 40%, respectively. Level 1 automated cardiac function post-processing is fast and highly reproducible with varying accuracy. Level 2 automation balances speed and accuracy.