Why rankings of biomedical image analysis competitions should be interpreted with care

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Proton range shift analysis on brain pseudo-CT generated from T1 and T2 MR

BACKGROUND

In radiotherapy, MR imaging is only used because it has significantly better soft tissue contrast than CT, but it lacks electron density information needed for dose calculation. This work assesses the feasibility of using pseudo-CT (pCT) generated from T1w/T2w MR for proton treatment planning, where proton range comparisons are performed between standard CT and pCT.

MATERIAL AND METHODS

MR and CT data from 14 glioblastoma patients were used in this study. The pCT was generated by using conversion libraries obtained from tissue segmentation and anatomical regioning of the T1w/T2w MR. For each patient, a plan consisting of three 18 Gy beams was designed on the pCT, for a total of 42 analyzed beams. The plan was then transferred onto the CT that represented the ground truth. Range shift (RS) between pCT and CT was computed at R₈₀ over 10 slices. The acceptance threshold for RS was according to clinical guidelines of two institutions. A γ-index test was also performed on the total dose for each patient.

RESULTS

Mean absolute error and bias for the pCT were 124 ± 10 and -16 ± 26 Hounsfield Units (HU), respectively. The median and interquartile range of RS was 0.5 and 1.4 mm, with highest absolute value being 4.4 mm. Of the 42 beams, 40 showed RS less than the clinical range margin. The two beams with larger RS were both in the cranio-caudal direction and had segmentation errors due to the partial volume effect, leading to misassignment of the HU.

CONCLUSIONS

This study showed the feasibility of using T1w and T2w MRI to generate a pCT for proton therapy treatment, thus avoiding the use of a planning CT and allowing better target definition and possibilities for online adaptive therapies. Further improvements of the methodology are still required to improve the conversion from MRI intensities to HUs.

Autosegmentation for thoracic radiation treatment planning: A grand challenge at AAPM 2017

PURPOSE

This report presents the methods and results of the Thoracic Auto-Segmentation Challenge organized at the 2017 Annual Meeting of American Association of Physicists in Medicine. The purpose of the challenge was to provide a benchmark dataset and platform for evaluating performance of autosegmentation methods of organs at risk (OARs) in thoracic CT images.

METHODS

Sixty thoracic CT scans provided by three different institutions were separated into 36 training, 12 offline testing, and 12 online testing scans. Eleven participants completed the offline challenge, and seven completed the online challenge. The OARs were left and right lungs, heart, esophagus, and spinal cord. Clinical contours used for treatment planning were quality checked and edited to adhere to the RTOG 1106 contouring guidelines. Algorithms were evaluated using the Dice coefficient, Hausdorff distance, and mean surface distance. A consolidated score was computed by normalizing the metrics against interrater variability and averaging over all patients and structures.

RESULTS

The interrater study revealed highest variability in Dice for the esophagus and spinal cord, and in surface distances for lungs and heart. Five out of seven algorithms that participated in the online challenge employed deep-learning methods. Although the top three participants using deep learning produced the best segmentation for all structures, there was no significant difference in the performance among them. The fourth place participant used a multi-atlas-based approach. The highest Dice scores were produced for lungs, with averages ranging from 0.95 to 0.98, while the lowest Dice scores were produced for esophagus, with a range of 0.55-0.72.

CONCLUSION

The results of the challenge showed that the lungs and heart can be segmented fairly accurately by various algorithms, while deep-learning methods performed better on the esophagus. Our dataset together with the manual contours for all training cases continues to be available publicly as an ongoing benchmarking resource.

Subject-specific Brain Tumor Growth Modelling via An Efficient Bayesian Inference Framework

An accurate prediction of brain tumor progression is crucial for optimized treatment of the tumors. Gliomas are primarily treated by combining surgery, external beam radiotherapy, and chemotherapy. Among them, radiotherapy is a non-invasive and effective therapy, and an understanding of tumor growth will allow better therapy planning. In particular, estimating parameters associated with tumor growth, such as the diffusion coefficient and proliferation rate, is crucial to accurately characterize physiology of tumor growth and to develop predictive models of tumor infiltration and recurrence. Accurate parameter estimation, however, is a challenging task due to inaccurate tumor boundaries and the approximation of the tumor growth model. Here, we introduce a Bayesian framework for a subject-specific tumor growth model that estimates the tumor parameters effectively. This is achieved by using an improved elliptical slice sampling method based on an adaptive sample region. Experimental results on clinical data demonstrate that the proposed method provides a higher acceptance rate, while preserving the parameter estimation accuracy, compared with other state-of-the-art methods such as Metropolis-Hastings and elliptical slice sampling without any modification. Our approach has the potential to provide a method to individualize therapy, thereby offering an optimized treatment.

Impact of interfractional motion on hypofractionated pencil beam scanning proton therapy and VMAT delivery for prostate cancer

PURPOSE

Hypofractionated radiotherapy of prostate cancer is gaining clinical acceptance given its potential increase in therapeutic ratio and evidence for noninferiority and lack of added late toxicities compared to conventional fractionation. However, concerns have been raised that smaller number of fractions might lead to larger dosimetric influence by interfractional motion. We aim to compare the effect of these variations on hypofractionated pencil beam scanning (PBS) proton therapy and volumetric modulated arc therapy (VMAT) for localized prostate cancer.

METHODS

Weekly CT images were acquired for 6 patients participating in a randomized clinical trial. PBS plans featuring bilateral (BL) and a combination of lateral and anterior-oblique beams (AOL), and VMAT plans were created. All patients were treated to a conventional 79.2 Gy total dose in 44 fractions. For this study, hypofractionated dose to the prostate gland was 51.6 Gy in 12 fractions or 36.25 Gy in 5 fractions, and 32.8, and 23.1 Gy to proximal seminal vesicles, respectively. Patients were simulated with endorectal balloons to aid gland immobilization. Three fiducial markers were implanted for setup guidance. All plans were recomputed on the weekly CT images after aligning with the simulation CT. The entire set of 9 CT images was used for dose recalculation for 12-fraction and only 5 used for the 5-fraction case. Adaptive range adjustments were applied to anterior-oblique beams assuming clinical availability of in vivo range verification. Fractional doses were summed using deformable dose accumulation to approximate the delivered dose. Biologically equivalent dose to 2 Gy(EQD2) was calculated assuming α/β of 1.5 Gy for prostate and 3 Gy for bladder and rectum.

RESULTS

The median delivered prostate D98 was 0.13/0.14/0.13 Gy(EQD2) smaller than planned for PBS-BL, 0.13/0.27/0.17 Gy(EQD2) for PBS-AOL and 0.59/0.66/0.59 Gy(EQD2) for VMAT, for 44/12/5 fractions, respectively. The largest D98 reduction was 1.5 and 3.5 Gy(EQD2) for CTV1 and CTV2, respectively. Target dose degradation was comparable for all fractionation schemes within each modality. The maximum increase in rectum D2 was 0.98 Gy(EQD2) for a 5-fraction PBS case.

CONCLUSIONS

The robustness of PBS and VMAT were comparable for all patients for the studied fractionations. The delivered target dose generally remained within clinical tolerance and the deviations were relatively minor for both fractionation schemes. The delivered OAR dose stayed in compliance with the RTOG hypofractionation constraints for all cases.

Multi atlas based segmentation: should we prefer the best atlas group over the group of best atlases?

Multi atlas based segmentation (MABS) uses a database of atlas images, and an atlas selection process is used to choose an atlas subset for registration and voting. In the current state of the art, atlases are chosen according to a similarity criterion between the target subject and each atlas in the database. In this paper, we propose a new concept for atlas selection that relies on selecting the best performing group of atlases rather than the group of highest scoring individual atlases. Experiments were performed using CT images of 50 patients, with contours of brainstem and parotid glands. The dataset was randomly split into two groups: 20 volumes were used as an atlas database and 30 served as target subjects for testing. Classic oracle selection, where atlases are chosen by the highest dice similarity coefficient (DSC) with the target, was performed. This was compared to oracle group selection, where all the combinations of atlas subgroups were considered and scored by computing DSC with the target subject. Subsequently, convolutional neural networks were designed to predict the best group of atlases. The results were also compared with the selection strategy based on normalized mutual information (NMI). Oracle group was proven to be significantly better than classic oracle selection (p  <  10^-5). Atlas group selection led to a median  ±  interquartile DSC of 0.740  ±  0.084, 0.718  ±  0.086 and 0.670  ±  0.097 for brainstem and left/right parotid glands respectively, outperforming NMI selection 0.676  ±  0.113, 0.632  ±  0.104 and 0.606  ±  0.118 (p  <  0.001) as well as classic oracle selection. The implemented methodology is a proof of principle that selecting the atlases by considering the performance of the entire group of atlases instead of each single atlas leads to higher segmentation accuracy, being even better then current oracle strategy. This finding opens a new discussion about the most appropriate atlas selection criterion for MABS.

Evaluation of segmentation methods on head and neck CT: Auto-segmentation challenge 2015

PURPOSE

Automated delineation of structures and organs is a key step in medical imaging. However, due to the large number and diversity of structures and the large variety of segmentation algorithms, a consensus is lacking as to which automated segmentation method works best for certain applications. Segmentation challenges are a good approach for unbiased evaluation and comparison of segmentation algorithms.

METHODS

In this work, we describe and present the results of the Head and Neck Auto-Segmentation Challenge 2015, a satellite event at the Medical Image Computing and Computer Assisted Interventions (MICCAI) 2015 conference. Six teams participated in a challenge to segment nine structures in the head and neck region of CT images: brainstem, mandible, chiasm, bilateral optic nerves, bilateral parotid glands, and bilateral submandibular glands.

RESULTS

This paper presents the quantitative results of this challenge using multiple established error metrics and a well-defined ranking system. The strengths and weaknesses of the different auto-segmentation approaches are analyzed and discussed.

CONCLUSIONS

The Head and Neck Auto-Segmentation Challenge 2015 was a good opportunity to assess the current state-of-the-art in segmentation of organs at risk for radiotherapy treatment. Participating teams had the possibility to compare their approaches to other methods under unbiased and standardized circumstances. The results demonstrate a clear tendency toward more general purpose and fewer structure-specific segmentation algorithms.

Proton dose calculation on scatter-corrected CBCT image: Feasibility study for adaptive proton therapy

PURPOSE

To demonstrate the feasibility of proton dose calculation on scatter-corrected cone-beam computed tomographic (CBCT) images for the purpose of adaptive proton therapy.

METHODS

CBCT projection images were acquired from anthropomorphic phantoms and a prostate patient using an on-board imaging system of an Elekta infinity linear accelerator. Two previously introduced techniques were used to correct the scattered x-rays in the raw projection images: uniform scatter correction (CBCTus) and a priori CT-based scatter correction (CBCTap). CBCT images were reconstructed using a standard FDK algorithm and GPU-based reconstruction toolkit. Soft tissue ROI-based HU shifting was used to improve HU accuracy of the uncorrected CBCT images and CBCTus, while no HU change was applied to the CBCTap. The degree of equivalence of the corrected CBCT images with respect to the reference CT image (CTref) was evaluated by using angular profiles of water equivalent path length (WEPL) and passively scattered proton treatment plans. The CBCTap was further evaluated in more realistic scenarios such as rectal filling and weight loss to assess the effect of mismatched prior information on the corrected images.

RESULTS

The uncorrected CBCT and CBCTus images demonstrated substantial WEPL discrepancies (7.3 ± 5.3 mm and 11.1 ± 6.6 mm, respectively) with respect to the CTref, while the CBCTap images showed substantially reduced WEPL errors (2.4 ± 2.0 mm). Similarly, the CBCTap-based treatment plans demonstrated a high pass rate (96.0% ± 2.5% in 2 mm/2% criteria) in a 3D gamma analysis.

CONCLUSIONS

A priori CT-based scatter correction technique was shown to be promising for adaptive proton therapy, as it achieved equivalent proton dose distributions and water equivalent path lengths compared to those of a reference CT in a selection of anthropomorphic phantoms.

Investigation of cone-beam CT image quality trade-off for image-guided radiation therapy

It is well-known that projections acquired over an angular range slightly over 180° (so-called short scan) are sufficient for fan-beam reconstruction. However, due to practical imaging conditions (projection data and reconstruction image discretization, physical factors, and data noise), the short-scan reconstructions may have different appearances and properties from the full-scan (scans over 360°) reconstructions. Nevertheless, short-scan configurations have been used in applications such as cone-beam CT (CBCT) for head-neck-cancer image-guided radiation therapy (IGRT) that only requires a small field of view due to the potential reduced imaging time and dose. In this work, we studied the image quality trade-off for full, short, and full/short scan configurations with both conventional filtered-backprojection (FBP) reconstruction and iterative reconstruction algorithms based on total-variation (TV) minimization for head-neck-cancer IGRT. Anthropomorphic and Catphan phantoms were scanned at different exposure levels with a clinical scanner used in IGRT. Both visualization- and numerical-metric-based evaluation studies were performed. The results indicate that the optimal exposure level and number of views are in the middle range for both FBP and TV-based iterative algorithms and the optimization is object-dependent and task-dependent. The optimal view numbers decrease with the total exposure levels for both FBP and TV-based algorithms. The results also indicate there are slight differences between FBP and TV-based iterative algorithms for the image quality trade-off: FBP seems to be more in favor of larger number of views while the TV-based algorithm is more robust to different data conditions (number of views and exposure levels) than the FBP algorithm. The studies can provide a general guideline for image-quality optimization for CBCT used in IGRT and other applications.

Validation of a deformable image registration technique for cone beam CT-based dose verification

PURPOSE

As radiation therapy evolves toward more adaptive techniques, image guidance plays an increasingly important role, not only in patient setup but also in monitoring the delivered dose and adapting the treatment to patient changes. This study aimed to validate a method for evaluation of delivered intensity modulated radiotherapy (IMRT) dose based on multimodal deformable image registration (dir) for prostate treatments.

METHODS

A pelvic phantom was scanned with CT and cone-beam computed tomography (CBCT). Both images were digitally deformed using two realistic patient-based deformation fields. The original CT was then registered to the deformed CBCT resulting in a secondary deformed CT. The registration quality was assessed as the ability of the dir method to recover the artificially induced deformations. The primary and secondary deformed CT images as well as vector fields were compared to evaluate the efficacy of the registration method and it's suitability to be used for dose calculation. plastimatch, a free and open source software was used for deformable image registration. A B-spline algorithm with optimized parameters was used to achieve the best registration quality. Geometric image evaluation was performed through voxel-based Hounsfield unit (HU) and vector field comparison. For dosimetric evaluation, IMRT treatment plans were created and optimized on the original CT image and recomputed on the two warped images to be compared. The dose volume histograms were compared for the warped structures that were identical in both warped images. This procedure was repeated for the phantom with full, half full, and empty bladder.

RESULTS

The results indicated mean HU differences of up to 120 between registered and ground-truth deformed CT images. However, when the CBCT intensities were calibrated using a region of interest (ROI)-based calibration curve, these differences were reduced by up to 60%. Similarly, the mean differences in average vector field lengths decreased from 10.1 to 2.5 mm when CBCT was calibrated prior to registration. The results showed no dependence on the level of bladder filling. In comparison with the dose calculated on the primary deformed CT, differences in mean dose averaged over all organs were 0.2% and 3.9% for dose calculated on the secondary deformed CT with and without CBCT calibration, respectively, and 0.5% for dose calculated directly on the calibrated CBCT, for the full-bladder scenario. Gamma analysis for the distance to agreement of 2 mm and 2% of prescribed dose indicated a pass rate of 100% for both cases involving calibrated CBCT and on average 86% without CBCT calibration.

CONCLUSIONS

Using deformable registration on the planning CT images to evaluate the IMRT dose based on daily CBCTs was found feasible. The proposed method will provide an accurate dose distribution using planning CT and pretreatment CBCT data, avoiding the additional uncertainties introduced by CBCT inhomogeneity and artifacts. This is a necessary initial step toward future image-guided adaptive radiotherapy of the prostate.

Radiological Protection in Cone Beam Computed Tomography (CBCT). ICRP Publication 129

The objective of this publication is to provide guidance on radiological protection in the new technology of cone beam computed tomography (CBCT). Publications 87 and 102 dealt with patient dose management in computed tomography (CT) and multi-detector CT. The new applications of CBCT and the associated radiological protection issues are substantially different from those of conventional CT. The perception that CBCT involves lower doses was only true in initial applications. CBCT is now used widely by specialists who have little or no training in radiological protection. This publication provides recommendations on radiation dose management directed at different stakeholders, and covers principles of radiological protection, training, and quality assurance aspects. Advice on appropriate use of CBCT needs to be made widely available. Advice on optimisation of protection when using CBCT equipment needs to be strengthened, particularly with respect to the use of newer features of the equipment. Manufacturers should standardise radiation dose displays on CBCT equipment to assist users in optimisation of protection and comparisons of performance. Additional challenges to radiological protection are introduced when CBCT-capable equipment is used for both fluoroscopy and tomography during the same procedure. Standardised methods need to be established for tracking and reporting of patient radiation doses from these procedures. The recommendations provided in this publication may evolve in the future as CBCT equipment and applications evolve. As with previous ICRP publications, the Commission hopes that imaging professionals, medical physicists, and manufacturers will use the guidelines and recommendations provided in this publication for implementation of the Commission's principle of optimisation of protection of patients and medical workers, with the objective of keeping exposures as low as reasonably achievable, taking into account economic and societal factors, and consistent with achieving the necessary medical outcomes.

Gain Correction for an X-ray Imaging System With a Movable Flat Panel Detector and Intrinsic Localization Crosshair

Gain calibration for X-ray imaging systems with a movable flat panel detector and an intrinsic crosshair is a challenge due to the geometry-dependent heel effect and crosshair artifact. This study aims to develop a gain correction method for such systems by implementing the Multi-Acquisition Gain Image Correction technique. Flood field images containing crosshair and heel effect were acquired in 4 different flat panel detector positions at fixed exposure parameters. The crosshair region was automatically detected using common image processing algorithms and removed by a simple interpolation procedure, resulting in a crosshair-removed image. A large kernel-based correction was then used to remove the heel effect. Mask filters corresponding to each crosshair region were applied to the resultant heel effect-removed images to invalidate the pixels of the original crosshair region. Finally, a seamless gain map was composed with corresponding valid pixels from the processed images either by the sequential replacement or by the selective averaging techniques developed in this study. Quantitative evaluation was performed based on normalized noise power spectrum and detective quantum efficiency improvement factor for the flood field images corrected by the Multi-Acquisition Gain Image Correction-based gain maps. For comparison purposes, a single crosshair-removed gain map was also tested. As a result, it was demonstrated that the Multi-Acquisition Gain Image Correction technique achieved better image quality than the crosshair-removed technique, showing lower normalized noise power spectrum values over most of spatial frequencies. The improvement was more obvious at the priori-crosshair region of the gain map. The mean detective quantum efficiency improvement factor was 1.09 ± 0.06, 2.46 ± 0.32, and 3.34 ± 0.36 in the priori-crosshair region and 2.35 ± 0.31, 2.33 ± 0.31, and 3.09 ± 0.34 in the normal region, for crosshair-removed, Multi-Acquisition Gain Image Correction-sequential replacement, and Multi-Acquisition Gain Image Correction-selective averaging techniques, respectively. Therefore, this study indicates that the introduced Multi-Acquisition Gain Image Correction technique is an appropriate method for gain calibration of an imaging system associated with a moving flat panel detector and an intrinsic crosshair.

Stopping Criteria for Log-Domain Diffeomorphic Demons Registration: An Experimental Survey for Radiotherapy Application

A crucial issue in deformable image registration is achieving a robust registration algorithm at a reasonable computational cost. Given the iterative nature of the optimization procedure an algorithm must automatically detect convergence, and stop the iterative process when most appropriate. This paper ranks the performances of three stopping criteria and six stopping value computation strategies for a Log-Domain Demons Deformable registration method simulating both a coarse and a fine registration. The analyzed stopping criteria are: (a) velocity field update magnitude, (b) mean squared error, and (c) harmonic energy. Each stoping condition is formulated so that the user defines a threshold ∊, which quantifies the residual error that is acceptable for the particular problem and calculation strategy. In this work, we did not aim at assigning a value to e, but to give insights in how to evaluate and to set the threshold on a given exit strategy in a very popular registration scheme. Experiments on phantom and patient data demonstrate that comparing the optimization metric minimum over the most recent three iterations with the minimum over the fourth to sixth most recent iterations can be an appropriate algorithm stopping strategy. The harmonic energy was found to provide best trade-off between robustness and speed of convergence for the analyzed registration method at coarse registration, but was outperformed by mean squared error when all the original pixel information is used. This suggests the need of developing mathematically sound new convergence criteria in which both image and vector field information could be used to detect the actual convergence, which could be especially useful when considering multi-resolution registrations. Further work should be also dedicated to study same strategies performances in other deformable registration methods and body districts.

Computing proton dose to irregularly moving targets

PURPOSE

While four-dimensional computed tomography (4DCT) and deformable registration can be used to assess the dose delivered to regularly moving targets, there are few methods available for irregularly moving targets. 4DCT captures an idealized waveform, but human respiration during treatment is characterized by gradual baseline shifts and other deviations from a periodic signal. This paper describes a method for computing the dose delivered to irregularly moving targets based on 1D or 3D waveforms captured at the time of delivery.

METHODS

The procedure uses CT or 4DCT images for dose calculation, and 1D or 3D respiratory waveforms of the target position at time of delivery. Dose volumes are converted from their Cartesian geometry into a beam-specific radiological depth space, parameterized in 2D by the beam aperture, and longitudinally by the radiological depth. In this new frame of reference, the proton doses are translated according to the motion found in the 1D or 3D trajectory. These translated dose volumes are weighted and summed, then transformed back into Cartesian space, yielding an estimate of the dose that includes the effect of the measured breathing motion. The method was validated using a synthetic lung phantom and a single representative patient CT. Simulated 4DCT was generated for the phantom with 2 cm peak-to-peak motion.

RESULTS

A passively-scattered proton treatment plan was generated using 6 mm and 5 mm smearing for the phantom and patient plans, respectively. The method was tested without motion, and with two simulated breathing signals: a 2 cm amplitude sinusoid, and a 2 cm amplitude sinusoid with 3 cm linear drift in the phantom. The tumor positions were equally weighted for the patient calculation. Motion-corrected dose was computed based on the mid-ventilation CT image in the phantom and the peak exhale position in the patient. Gamma evaluation was 97.8% without motion, 95.7% for 2 cm sinusoidal motion, 95.7% with 3 cm drift in the phantom (2 mm, 2%), and 90.8% (3 mm, 3%)for the patient data.

CONCLUSIONS

We have demonstrated a method for accurately reproducing proton dose to an irregularly moving target from a single CT image. We believe this algorithm could prove a useful tool to study the dosimetric impact of baseline shifts either before or during treatment.