Interobserver and intraobserver variability in the assessment of pulmonary nodule size on CT using film and computer display methods

Comparability of Pulmonary Nodule Size Measurements among Different Scanners and Protocols: Should Diameter Be Favorized over Volume?

BACKGROUND

To assess the impact of the lung cancer screening protocol recommended by the European Society of Thoracic Imaging (ESTI) on nodule diameter, volume, and density throughout different computed tomography (CT) scanners.

METHODS

An anthropomorphic chest phantom containing fourteen different-sized (range 3-12 mm) and CT-attenuated (100 HU, -630 HU and -800 HU, termed as solid, GG1 and GG2) pulmonary nodules was imaged on five CT scanners with institute-specific standard protocols (P_S) and the lung cancer screening protocol recommended by ESTI (ESTI protocol, P_E). Images were reconstructed with filtered back projection (FBP) and iterative reconstruction (REC). Image noise, nodule density and size (diameter/volume) were measured. Absolute percentage errors (APEs) of measurements were calculated.

RESULTS

Using P_E, dosage variance between different scanners tended to decrease compared to P_S, and the mean differences were statistically insignificant (p = 0.48). P_S and P_E(REC) showed significantly less image noise than P_E(FBP) (p < 0.001). The smallest size measurement errors were noted with volumetric measurements in P_E(REC) and highest with diametric measurements in P_E(FBP). Volume performed better than diameter measurements in solid and GG1 nodules (p < 0.001). However, in GG2 nodules, this could not be observed (p = 0.20). Regarding nodule density, REC values were more consistent throughout different scanners and protocols.

CONCLUSION

Considering radiation dose, image noise, nodule size, and density measurements, we fully endorse the ESTI screening protocol including the use of REC. For size measurements, volume should be preferred over diameter.

The clinical impact of observer variability in lung nodule classification in children with Wilms tumour

OBJECTIVES

To investigate the extent to which observer variability of computed tomography (CT) lung nodule assessment may affect clinical treatment stratification in Wilms tumour (WT) patients, according to the recent Société Internationale d'Oncologie Pédiatrique Renal Tumour Study Group (SIOP-RTSG) UMBRELLA protocol.

METHODS

I: CT thoraces of children with WT submitted for central review were used to estimate size distribution of lung metastases. II: Scans were selected for blinded review by five radiologists to determine intra- and inter-observer variability. They assessed identical scans on two occasions 6 months apart. III: Monte Carlo simulation (MCMC) was used to predict the clinical impact of observer variation when applying the UMBRELLA protocol size criteria.

RESULTS

Lung nodules were found in 84 out of 360 (23%) children with WT. For 21 identified lung nodules, inter-observer limits of agreement (LOA) for the five readers were ±2.4 and ±1.4 mm (AP diameter), ±1.9 and ±1.8 mm (TS diameter) and ±2.0 and ±2.4 mm (LS diameter) at assessments 1 and 2. Intra-observer LOA across the three dimensions were ±1.5, ±2.2, ±3.5, ±3.1 and ±2.6 mm (readers 1-5). MCMC demonstrated that 17% of the patients with a 'true' nodule size of ≥3 mm will be scored as <3 mm, and 21% of the patients with a 'true' nodule size of <3 mm will be scored as being ≥3 mm.

CONCLUSION

A significant intra-inter observer variation was found when measuring lung nodules on CT for patients with WT. This may have significant implications on treatment stratification, and thereby outcome, when applying a threshold of ≥3 mm for a lung nodule to dictate metastatic status.

Intra- and Inter-Reader Variations in Lung Nodule Measurements: Influences of Nodule Size, Location, and Observers

(1) Background: Accurate measurement of lung-nodule size is necessary, but whether a three-dimensional volume measurement is better or more reliable than the one-dimensional method is still unclear. This study aimed to investigate the intra- and inter-reader variations according to nodule type, size, three-dimensional volume measurements, and one-dimensional linear measurements. (2) Methods: This retrospective study included computed tomography (CT) examinations of lung nodules and volume measurements performed from October to December 2016. Two radiologists independently performed all measurements. Intra-class correlation coefficients (ICC) and Bland-Altman plots were used for analysis. (3) Results: The overall variability in the calculated volume was larger than when using the semiautomatic volume measurement. Nodules <6 mm tended to have larger variability than nodules ≥6 mm in both one-dimensional and calculated volume measurements. The isolated type showed smaller variability in both intra- and inter-reader comparisons. The juxta-vascular type showed the largest variability in both one-dimensional and calculated volume measurements. The variability was decreased when using the 3D volume semiautomated software. (4) Conclusions: The present study suggests that 3D semiautomatic volume measurements showed lower variability than the calculated volume measurement. Nodule size and location influence measurement variability. The intra- and inter-reader variabilities in nodule volume measurement were considerable.

A unified framework for simultaneous assessment of accuracy, between-, and within-reader variability of image segmentations

Medical imaging is utilized in a wide range of clinical applications. To enable a detailed quantitative analysis, medical images must often be segmented to label (delineate) structures of interest; for example, a tumor. Frequently, manual segmentation is utilized in clinical practice (e.g., in radiation oncology) to define such structures of interest. However, it can be quite time consuming and subject to substantial between-, and within-reader variability. A more reproducible, less variable, and more time efficient segmentation approach is likely to improve medical treatment. This potential has spurred the development of segmentation algorithms which harness computational power. Segmentation algorithms’ widespread use is limited due to difficulty in quantifying their performance relative to manual segmentation, which itself is subject to variation. This paper presents a statistical model which simultaneously estimates segmentation method accuracy, and between- and within-reader variability. The model is simultaneously fit for multiple segmentation methods within a unified Bayesian framework. The Bayesian model is compared to other methods used in literature via a simulation study, and application to head and neck cancer PET/CT data. The modeling framework is flexible and can be employed in numerous comparison applications. Several alternate applications are discussed in the paper.

"Rounding" the Size of Pulmonary Nodules: Impact of Rounding Methods on Nodule Management, as Defined by the 2017 Fleischner Society Guidelines

RATIONALE AND OBJECTIVES

The objective of this study was to quantify the impact of different rounding methods on size measurements of pulmonary nodules and to determine the number of nodules that change management categories as a result of rounding.

MATERIALS AND METHODS

For this retrospective institutional review board-approved study, we included 503 incidental pulmonary nodules (308 solid and 195 subsolid) from a data repository. Long and short axes were measured. Average diameters were calculated using four different rounding methods (method 1: no rounding; method 2: rounding only the average diameter to the closest millimeter; method 3: rounding only short and long axes; and method 4: rounding short and long axes and the average diameter to the closest millimeter). Nodules were classified for each rounding method according to the 2017 Fleischner Society guideline management categories. Measurements were compared among the four rounding methods using analysis of variance.

RESULTS

Without rounding, the average nodule diameter was 15.67 ± 5.97 mm. This increased between 0.03  and 0.29 mm using rounding methods 2-4 (range: P < 0.001-0.017). The nodule size was more frequently rounded up (range: 52.1%-77.5%) than rounded down (range: 17.7%-42.5%) using rounding methods 2-4, as compared to no rounding. In the 308 solid nodules, up to 2.9% of the nodules changed management category, whereas none of the 195 subsolid nodules changed category.

CONCLUSIONS

Rounding methods have a small absolute but statically significant effect on nodule size, impacting management category in less than 3% of the nodules. This suggests that, in clinical practice, any rounding method can be used for determining nodule size without substantially biasing individual nodules toward given management categories.

Toward Understanding the Size Dependence of Shape Features for Predicting Spiculation in Lung Nodules for Computer-Aided Diagnosis

We analyze the importance of shape features for predicting spiculation ratings assigned by radiologists to lung nodules in computed tomography (CT) scans. Using the Lung Image Database Consortium (LIDC) data and classification models based on decision trees, we demonstrate that the importance of several shape features increases disproportionately relative to other image features with increasing size of the nodule. Our shaped-based classification results show an area under the receiver operating characteristic (ROC) curve of 0.65 when classifying spiculation for small nodules and an area of 0.91 for large nodules, resulting in a 26% difference in classification performance using shape features. An analysis of the results illustrates that this change in performance is driven by features that measure boundary complexity, which perform well for large nodules but perform relatively poorly and do no better than other features for small nodules. For large nodules, the roughness of the segmented boundary maps well to the semantic concept of spiculation. For small nodules, measuring directly the complexity of hard segmentations does not yield good results for predicting spiculation due to limits imposed by spatial resolution and the uncertainty in boundary location. Therefore, a wider range of features, including shape, texture, and intensity features, are needed to predict spiculation ratings for small nodules. A further implication is that the efficacy of shape features for a particular classifier used to create computer-aided diagnosis systems depends on the distribution of nodule sizes in the training and testing sets, which may not be consistent across different research studies.

Algorithm Variability in the Estimation of Lung Nodule Volume From Phantom CT Scans: Results of the QIBA 3A Public Challenge

RATIONALE AND OBJECTIVES

Quantifying changes in lung tumor volume is important for diagnosis, therapy planning, and evaluation of response to therapy. The aim of this study was to assess the performance of multiple algorithms on a reference data set. The study was organized by the Quantitative Imaging Biomarker Alliance (QIBA).

MATERIALS AND METHODS

The study was organized as a public challenge. Computed tomography scans of synthetic lung tumors in an anthropomorphic phantom were acquired by the Food and Drug Administration. Tumors varied in size, shape, and radiodensity. Participants applied their own semi-automated volume estimation algorithms that either did not allow or allowed post-segmentation correction (type 1 or 2, respectively). Statistical analysis of accuracy (percent bias) and precision (repeatability and reproducibility) was conducted across algorithms, as well as across nodule characteristics, slice thickness, and algorithm type.

RESULTS

Eighty-four percent of volume measurements of QIBA-compliant tumors were within 15% of the true volume, ranging from 66% to 93% across algorithms, compared to 61% of volume measurements for all tumors (ranging from 37% to 84%). Algorithm type did not affect bias substantially; however, it was an important factor in measurement precision. Algorithm precision was notably better as tumor size increased, worse for irregularly shaped tumors, and on the average better for type 1 algorithms. Over all nodules meeting the QIBA Profile, precision, as measured by the repeatability coefficient, was 9.0% compared to 18.4% overall.

CONCLUSION

The results achieved in this study, using a heterogeneous set of measurement algorithms, support QIBA quantitative performance claims in terms of volume measurement repeatability for nodules meeting the QIBA Profile criteria.

The IASLC Lung Cancer Staging Project: Proposals for Coding T Categories for Subsolid Nodules and Assessment of Tumor Size in Part-Solid Tumors in the Forthcoming Eighth Edition of the TNM Classification of Lung Cancer

This article proposes codes for the primary tumor categories of adenocarcinoma in situ (AIS) and minimally invasive adenocarcinoma (MIA) and a uniform way to measure tumor size in part-solid tumors for the eighth edition of the tumor, node, and metastasis classification of lung cancer. In 2011, new entities of AIS, MIA, and lepidic predominant adenocarcinoma were defined, and they were later incorporated into the 2015 World Health Organization classification of lung cancer. To fit these entities into the T component of the staging system, the Tis category is proposed for AIS, with Tis (AIS) specified if it is to be distinguished from squamous cell carcinoma in situ (SCIS), which is to be designated Tis (SCIS). We also propose that MIA be classified as T1mi. Furthermore, the use of the invasive size for T descriptor size follows a recommendation made in three editions of the Union for International Cancer Control tumor, node, and metastasis supplement since 2003. For tumor size, the greatest dimension should be reported both clinically and pathologically. In nonmucinous lung adenocarcinomas, the computed tomography (CT) findings of ground glass versus solid opacities tend to correspond respectively to lepidic versus invasive patterns seen pathologically. However, this correlation is not absolute; so when CT features suggest nonmucinous AIS, MIA, and lepidic predominant adenocarcinoma, the suspected diagnosis and clinical staging should be regarded as a preliminary assessment that is subject to revision after pathologic evaluation of resected specimens. The ability to predict invasive versus noninvasive size on the basis of solid versus ground glass components is not applicable to mucinous AIS, MIA, or invasive mucinous adenocarcinomas because they generally show solid nodules or consolidation on CT.

Role of the Quantitative Imaging Biomarker Alliance in optimizing CT for the evaluation of lung cancer screen-detected nodules

The Quantitative Imaging Biomarker Alliance (QIBA) is a multidisciplinary consortium sponsored by the RSNA to define processes that enable the implementation and advancement of quantitative imaging methods described in a QIBA profile document that outlines the process to reliably and accurately measure imaging features. A QIBA profile includes factors such as technical (product-specific) standards, user activities, and relationship to a clinically meaningful metric, such as with nodule measurement in the course of CT screening for lung cancer. In this report, the authors describe how the QIBA approach is being applied to the measurement of small pulmonary nodules such as those found during low-dose CT-based lung cancer screening. All sources of variance with imaging measurement were defined for this process. Through a process of experimentation, literature review, and assembly of expert opinion, the strongest evidence was used to define how to best implement each step in the imaging acquisition and evaluation process. This systematic approach to implementing a quantitative imaging biomarker with standardized specifications for image acquisition and postprocessing for a specific quantitative measurement of a pulmonary nodule results in consistent performance characteristics of the measurement (eg, bias and variance). Implementation of the QIBA small nodule profile may allow more efficient and effective clinical management of the diagnostic workup of individuals found to have suspicious pulmonary nodules in the course of lung cancer screening evaluation.

Evaluation and management of pulmonary nodules: state-of-the-art and future perspectives

INTRODUCTION

The imaging evaluation of pulmonary nodules, often incidentally detected on imaging examinations performed for other clinical reasons, is a frequently encountered clinical circumstance. With advances in imaging modalities, both the detection and characterization of pulmonary nodules continue to evolve and improve.

AREAS COVERED

This article will review the imaging modalities used to detect and diagnose benign and malignant pulmonary nodules, with a focus on computed tomography (CT), which continues to be the mainstay for evaluation. The authors discuss recent advances in the lung nodule management, and an algorithm for the management of indeterminate pulmonary nodules.

EXPERT OPINION

There are set of criteria that define a benign nodule, the most important of which are the lack of temporal change for 2 years or more, and certain benign imaging criteria, including specific patterns of calcification or the presence of fat. Although some indeterminate pulmonary nodules are immediately actionable, generally those approaching 1 cm or larger in diameter, at which size the diagnostic accuracy of tools such as positron emission tomography (PET)/CT, single photon emission CT (SPECT) and biopsy techniques are sufficient to warrant their use. The majority of indeterminate pulmonary nodules are under 1 cm, for which serial CT examinations through at least 2 years for solid nodules and 3 years for ground-glass nodules, are used to demonstrate either benign biologic behavior or otherwise. The management of incidental pulmonary nodules involves a multidisciplinary approach in which radiology plays a pivotal role. Newer imaging and postprocessing techniques have made this a more accurate technique eliminating ambiguity and unnecessary follow-up.

Functional imaging biomarkers for assessing response to treatment in liver and lung metastases

Management of patients with metastatic cancer and development of new treatments rely on imaging to provide non-invasive biomarkers of tumour response and progression. The widely used size-based criteria have increasingly become inadequate where early measures of response are required to avoid toxicity of ineffective treatments, as biological, physiologic, and molecular modifications in tumours occur before changes in gross tumour size. A multiparametric approach with the current range of imaging techniques allows functional aspects of tumours to be simultaneously interrogated. Appropriate use of these imaging techniques and their timing in relation to the treatment schedule, particularly in the context of clinical trials, is fundamental. There is a lack of consensus regarding which imaging parameters are most informative for a particular disease site and the best time to image so that, despite an increasing body of literature, open questions on these aspects remain. In addition, standardization of these new parameters is required. This review summarizes the published literature over the last decade on functional and molecular imaging techniques in assessing treatment response in liver and lung metastases.

Noninvasive characterization of the histopathologic features of pulmonary nodules of the lung adenocarcinoma spectrum using computer-aided nodule assessment and risk yield (CANARY)--a pilot study

INTRODUCTION

Pulmonary nodules of the adenocarcinoma spectrum are characterized by distinctive morphological and radiologic features and variable prognosis. Noninvasive high-resolution computed tomography-based risk stratification tools are needed to individualize their management.

METHODS

Radiologic measurements of histopathologic tissue invasion were developed in a training set of 54 pulmonary nodules of the adenocarcinoma spectrum and validated in 86 consecutively resected nodules. Nodules were isolated and characterized by computer-aided analysis, and data were analyzed by Spearman correlation, sensitivity, and specificity and the positive and negative predictive values.

RESULTS

Computer-aided nodule assessment and risk yield (CANARY) can noninvasively characterize pulmonary nodules of the adenocarcinoma spectrum. Unsupervised clustering analysis of high-resolution computed tomography data identified nine unique exemplars representing the basic radiologic building blocks of these lesions. The exemplar distribution within each nodule correlated well with the proportion of histologic tissue invasion, Spearman R = 0.87, p < 0.0001 and 0.89 and p < 0.0001 for the training and the validation set, respectively. Clustering of the exemplars in three-dimensional space corresponding to tissue invasion and lepidic growth was used to develop a CANARY decision algorithm that successfully categorized these pulmonary nodules as "aggressive" (invasive adenocarcinoma) or "indolent" (adenocarcinoma in situ and minimally invasive adenocarcinoma). Sensitivity, specificity, positive predictive value, and negative predictive value of this approach for the detection of aggressive lesions were 95.4, 96.8, 95.4, and 96.8%, respectively, in the training set and 98.7, 63.6, 94.9, and 87.5%, respectively, in the validation set.

CONCLUSION

CANARY represents a promising tool to noninvasively risk stratify pulmonary nodules of the adenocarcinoma spectrum.

Measurement methods and algorithms for the management of solid nodules

The increased detection of incidental small pulmonary nodules on multidetector computed tomography has driven attempts to refine the characterization and management of such nodules. A variety of methods have been developed to measure the size and biological activity of nodules to help define their nature, but these have limitations. Several clinical trials have assessed the efficacy of low-dose computed tomography screening for lung cancer and offer some insights into these limitations; however, they also provide evidence that refines existing nodule management strategies. This article reviews the size-based and functional measurement methods that can be used to predict the likelihood of malignancy in noncalcified solid pulmonary nodules and discusses their incorporation into existing algorithms for nodule management. The issue of multiple nodules and the optimum frequency and duration of follow-up are explored.

The use of tumour volumetrics to assess response to therapy in anticancer clinical trials

Serial evaluations of tumour burden using imaging, mainly computed tomography and magnetic resonance imaging, form the basis for assessing treatment response in many clinical trials of anticancer therapeutics. Traditionally, these evaluations have been based on linear measurements of tumour size. Such measurements have limitations related to variability in technical factors, tumour morphology and reader decisions. Measurements of entire tumour volumes may overcome some of the limitations of linear tumour measurements, improving our ability to detect small changes reliably and increasing statistical power per subject in a trial. Certain technical factors are known to affect the accuracy and precision of volume measurements, and work is in progress to define these factors more thoroughly and to qualify tumour volume as a biomarker for the purposes of drug development.

Computer-aided nodule detection and volumetry to reduce variability between radiologists in the interpretation of lung nodules at low-dose screening computed tomography

OBJECTIVE

The aim of this study was to evaluate whether a computer-aided diagnosis (CAD) system improves interobserver agreement in the interpretation of lung nodules at low-dose computed tomography (CT) screening for lung cancer.

MATERIALS AND METHODS

Baseline low-dose screening CT examinations from 134 participants enrolled in the National Lung Screening Trial were reviewed by 7 chest radiologists. All participants consented to the use of their deidentified images for research purposes. Screening results were classified as positive when noncalcified nodules larger than 4 mm in diameter were present. Follow-up evaluation was recommended according to the nodule diameter: 4 mm or smaller, more than 4 to 8 mm, and larger than 8 mm. When multiple nodules were present, recommendations were based on the largest nodule. Readers initially assessed the nodule presence visually and measured the average nodule diameter manually. Revision of their decisions after reviewing the CAD marks and size measurement was allowed. Interobserver agreement evaluated using multirater κ statistics was compared between initial assessment and that with CAD.

RESULTS

Multirater κ values for the positivity of the screening results and follow-up recommendations were improved from moderate (κ = 0.53-0.54) at initial assessment to good (κ = 0.66-0.67) after reviewing CAD results. The average percentage of agreement between reader pairs on the positivity of screening results and follow-up recommendations per case was also increased from 77% and 72% at initial assessment to 84% and 80% with CAD, respectively.

CONCLUSION

Computer-aided diagnosis may improve the reader agreement on the positivity of screening results and follow-up recommendations in the assessment of low-dose screening CT.

Measurement of tumor volumes improves RECIST-based response assessments in advanced lung cancer

OBJECTIVE

This study was designed to characterize the reproducibility of measurement for tumor volumes and their longest tumor diameters (LDs) and estimate the potential impact of using changes in tumor volumes instead of LDs as the basis for response assessments.

METHODS

We studied patients with advanced lung cancer who have been observed longitudinally with x-ray computed tomography in a multinational trial. A total of 71 time points from 10 patients with 13 morphologically complex target lesions were analyzed. A total of 6461 volume measurements and their corresponding LDs were made by seven independent teams using their own work flows and image analysis tools. Interteam agreement and overall interrater concurrence were characterized.

RESULTS

Interteam agreement between volume measurements was better than between LD measurements (ı = 0.945 vs 0.734, P = .005). The variability in determining the nadir was lower for volumes than for LDs (P = .005). Use of standard thresholds for the RECIST-based method and use of experimentally determined cutoffs for categorizing responses showed that volume measurements had a significantly greater sensitivity for detecting partial responses and disease progression. Earlier detection of progression would have led to earlier changes in patient management in most cases.

CONCLUSIONS

Our findings indicate that measurement of changes in tumor volumes is adequately reproducible. Using tumor volumes as the basis for response assessments could have a positive impact on both patient management and clinical trials. More authoritative work to qualify or discard changes in volume as the basis for response assessments should proceed.

Interobserver variability in chest CT and whole body FDG-PET screening for distant metastases in head and neck cancer patients

Purpose

The aim of the study was to assess the interobserver variability in chest computed tomography (CT) and whole body 2-deoxy-2-[¹⁸F]fluoro-d-glucose positron emission tomography (FDG-PET) screening for distant metastases in head and neck squamous cell carcinoma (HNSCC) patients.

Procedure

Chest CT and whole body FDG-PET of 69 HNSCC patients with high-risk factors who underwent screening for distant metastases were analyzed. All scans were independently read by two experienced radiologists or nuclear physicians who were blinded to the other examinations and follow-up results.

Results

A kappa of 0.516 was found for assessment of size on CT. Kappa values for origin and susceptibility of 0.406 and 0.512 for CT and 0.834 and 0.939 for PET were found, respectively. The overall conclusions had a kappa of 0.517–0.634 for CT and 0.820–1.000 for PET.

Conclusions

In screening for distant metastases in HNSCC patients with high-risk factors, chest CT readings had a reasonable to substantial agreement, while PET readings showed an almost perfect agreement. These findings suggest that for optimal assessment in clinical practice, PET most often can be scored by one observer, but CT should probably more often be scored by different observers in consensus or combined with PET.

X-ray computed tomography: semiautomated volumetric analysis of late-stage lung tumors as a basis for response assessments

Background. This study presents a semiautomated approach for volumetric analysis of lung tumors and evaluates the feasibility of using volumes as an alternative to line lengths as a basis for response evaluation criteria in solid tumors (RECIST). The overall goal for the implementation was to accurately, precisely, and efficiently enable the analyses of lesions in the lung under the guidance of an operator. Methods. An anthropomorphic phantom with embedded model masses and 71 time points in 10 clinical cases with advanced lung cancer was analyzed using a semi-automated workflow. The implementation was done using the Cognition Network Technology. Results. Analysis of the phantom showed an average accuracy of 97%. The analyses of the clinical cases showed both intra- and interreader variabilities of approximately 5% on average with an upper 95% confidence interval of 14% and 19%, respectively. Compared to line lengths, the use of volumes clearly shows enhanced sensitivity with respect to determining response to therapy. Conclusions. It is feasible to perform volumetric analysis efficiently with high accuracy and low variability, even in patients with late-stage cancer who have complex lesions.

Evaluation of reader variability in the interpretation of follow-up CT scans at lung cancer screening

PURPOSE

To measure reader agreement in determining whether lung nodules detected at baseline screening computed tomography (CT) had changed at subsequent screening examinations and to evaluate the variability in recommendations for further follow-up.

MATERIALS AND METHODS

All subjects were enrolled in the National Lung Screening Trial (NLST), and each participant consented to the use of their de-identified images for research purposes. The authors randomly selected 100 cases of nodules measuring at least 4.0 mm at 1-year screening CT that were considered by the original screening CT reader to be present on baseline CT scans; nodules considered by the original reader to have changed were oversampled. Selected images from each case showing the entire nodule at both examinations were preloaded on a picture archiving and communication system workstation. Nine radiologists served as readers, and they evaluated whether the nodule was present at baseline and recorded the bidimensional measurements and nodule characteristics at each examination, presence or absence of change, results of screening CT, and follow-up recommendations (high-level follow-up, low-level follow-up, no follow-up).

RESULTS

On the basis of reviews during case selection, five nodules seen at follow-up were judged not to have been present at baseline; for 19 of the remaining 95 cases, at least one reader judged the nodule not to have been present at baseline. For the 76 nodules that were unanimously considered to have been present at baseline, 21%-47% (mean ± standard deviation, 30% ± 9) were judged to have grown. The κ values were similar for growth (κ = 0.55) and a positive screening result (κ = 0.51) and were lower for a change in margins and attenuation (κ = 0.27-0.31). The κ value in the recommendation of high- versus low-level follow-up was high (κ = 0.66).

CONCLUSION

Reader agreement on nodule growth and screening result was moderate to substantial. Agreement on follow-up recommendations was lower.

Data sets for the qualification of volumetric CT as a quantitative imaging biomarker in lung cancer

The drug development industry is faced with increasing costs and decreasing success rates. New ways to understand biology as well as the increasing interest in personalized treatments for smaller patient segments requires new capabilities for the rapid assessment of treatment responses. Deployment of qualified imaging biomarkers lags apparent technology capabilities. The lack of consensus methods and qualification evidence needed for large-scale multi-center trials, as well as the standardization that allows them, are widely acknowledged to be the limiting factors. The current fragmentation in imaging vendor offerings, coupled with the independent activities of individual biopharmaceutical companies and their contract research organizations (CROs), may stand in the way of the greater opportunity were these efforts to be drawn together. A preliminary report, "Volumetric CT: a potential biomarker of response," of the Quantitative Imaging Biomarkers Alliance (QIBA) activity was presented at the Medical Imaging Continuum: Path Forward for Advancing the Uses of Medical Imaging in the Development of New Biopharmaceutical Products meeting of the Extended Pharmaceutical Research and Manufacturers of America (PhRMA) Imaging Group sponsored by the Drug Information Agency (DIA) in October 2008. The clinical context in Lung Cancer and a methodology for approaching the qualification of volumetric CT as a biomarker has since been reported [Acad. Radiol. 17, 100-106, 107-115 (2010)]. This report reviews the effort to collect and utilize publicly available data sets to provide a transparent environment in which to pursue the qualification activities in such a way as to allow independent peer review and verification of results. This article focuses specifically on our role as stewards of image sets for developing new tools.

Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists' performance

RATIONALE AND OBJECTIVES

The aim of this study was to evaluate the effect of computer-aided diagnosis (CAD) on radiologists' estimates of the likelihood of malignancy of lung nodules on computed tomographic (CT) imaging.

METHODS AND MATERIALS

A total of 256 lung nodules (124 malignant, 132 benign) were retrospectively collected from the thoracic CT scans of 152 patients. An automated CAD system was developed to characterize and provide malignancy ratings for lung nodules on CT volumetric images. An observer study was conducted using receiver-operating characteristic analysis to evaluate the effect of CAD on radiologists' characterization of lung nodules. Six fellowship-trained thoracic radiologists served as readers. The readers rated the likelihood of malignancy on a scale of 0% to 100% and recommended appropriate action first without CAD and then with CAD. The observer ratings were analyzed using the Dorfman-Berbaum-Metz multireader, multicase method.

RESULTS

The CAD system achieved a test area under the receiver-operating characteristic curve (A(z)) of 0.857 +/- 0.023 using the perimeter, two nodule radii measures, two texture features, and two gradient field features. All six radiologists obtained improved performance with CAD. The average A(z) of the radiologists improved significantly (P < .01) from 0.833 (range, 0.817-0.847) to 0.853 (range, 0.834-0.887).

CONCLUSION

CAD has the potential to increase radiologists' accuracy in assessing the likelihood of malignancy of lung nodules on CT imaging.

Extracardiac findings in cardiac computed tomographic angiography in patients at low to intermediate risk for coronary artery disease

PURPOSE

To evaluate the prevalence, clinical significance, interobserver agreement, and follow-up of extracardiac findings on coronary computed tomographic angiography (CTA).

METHODS

A prospectively recruited cohort of 80 patients at low to intermediate risk of having coronary artery disease underwent CTA with field of view imaging from lung apices to upper abdomen. Two staff radiologists read each scan independently. Scans read by reader no. 1 were read as part of routine clinical practice, and the findings were subsequently reclassified to potentially significant, as defined by requiring clinical or radiologic follow-up, and insignificant by a separate observer, whereas reader no. 2 retrospectively read and autonomously classified the findings as potentially significant or insignificant.

RESULTS

Reader no. 1 found 7 potentially significant findings in 7 patients and 33 insignificant findings in 29 patients. Reader no. 2 found 10 potentially significant findings in 10 patients and 59 insignificant findings in 42 patients. Inter-rater agreement was moderate (kappa = 0.49; 95% confidence interval, 0.31-0.67) for the presence vs the absence of extracardiac findings and moderate (kappa = 0.52; 95% confidence interval, 0.15-0.89) for the presence of potentially significant extracardiac findings. The most common potentially significant finding was possibly malignant lung nodule (n = 6 [reader 1], 4 [reader 2]). Four patients with potentially significant findings received follow-up imaging, and 1 patient underwent biopsy, which was complicated by pneumothorax. No diagnoses of malignancy were made.

CONCLUSIONS

Extracardiac findings are frequent and moderately reproducible, however, in this study, not associated with clinical benefit. Large prospective studies are required to establish whether reporting of extracardiac findings is associated with improved patient outcomes.

Quantitative imaging to assess tumor response to therapy: common themes of measurement, truth data, and error sources

RATIONALE

Early detection of tumor response to therapy is a key goal. Finding measurement algorithms capable of early detection of tumor response could individualize therapy treatment as well as reduce the cost of bringing new drugs to market. On an individual basis, the urgency arises from the desire to prevent continued treatment of the patient with a high-cost and/or high-risk regimen with no demonstrated individual benefit and rapidly switch the patient to an alternative efficacious therapy for that patient. In the context of bringing new drugs to market, such algorithms could demonstrate efficacy in much smaller populations, which would allow phase 3 trials to achieve statistically significant decisions with fewer subjects in shorter trials.

MATERIALS AND METHODS

This consensus-based article describes multiple, image modality-independent means to assess the relative performance of algorithms for measuring tumor change in response to therapy. In this setting, we describe specifically the example of measurement of tumor volume change from anatomic imaging as well as provide an overview of other promising generic analytic methods that can be used to assess change in heterogeneous tumors. To support assessment of the relative performance of algorithms for measuring small tumor change, data sources of truth are required.

RESULTS

Very short interval clinical imaging examinations and phantom scans provide known truth for comparative evaluation of algorithms.

CONCLUSIONS

For a given category of measurement methods, the algorithm that has the smallest measurement noise and least bias on average will perform best in early detection of true tumor change.

Noncalcified lung nodules: volumetric assessment with thoracic CT

Lung nodule volumetry is used for nodule diagnosis, as well as for monitoring tumor response to therapy. Volume measurement precision and accuracy depend on a number of factors, including image-acquisition and reconstruction parameters, nodule characteristics, and the performance of algorithms for nodule segmentation and volume estimation. The purpose of this article is to provide a review of published studies relevant to the computed tomographic (CT) volumetric analysis of lung nodules. A number of underexamined areas of research regarding volumetric accuracy are identified, including the measurement of nonsolid nodules, the effects of pitch and section overlap, and the effect of respiratory motion. The need for public databases of phantom scans, as well as of clinical data, is discussed. The review points to the need for continued research to examine volumetric accuracy as a function of a multitude of interrelated variables involved in the assessment of lung nodules. Understanding and quantifying the sources of volumetric measurement error in the assessment of lung nodules with CT would be a first step toward the development of methods to minimize that error through system improvements and to correctly account for any remaining error.

Cardiac computed tomographic angiography: evaluation of non-cardiac structures

Cardiac computed tomography is a promising new technology for non-invasive evaluation of the coronary arteries. As CT is inherently a high resolution volumetric imaging modality, data from structures other than the heart can be accessed in studies performed primarily for cardiac indications. Current generation scanners can easily detect abnormalities such as pulmonary emboli and aortic dissection on routine coronary CT angiograms. Many other abnormalities such as small pulmonary nodules can also be detected. While major abnormalities like aortic dissection are of obvious clinical importance, detection of incidental abnormalities such as small pulmonary nodules less than 4 mm in diameter has not yet been shown to positively affect patient outcomes, and may lead to unnecessary testing. Recommendations for image reconstruction and training in interpretation of incidental findings continue to evolve, but most agree that coronary CT angiography should be focused primarily on the coronary arteries.

Effect of CT scanning parameters on volumetric measurements of pulmonary nodules by 3D active contour segmentation: a phantom study

The purpose of this study is to investigate the effects of CT scanning and reconstruction parameters on automated segmentation and volumetric measurements of nodules in CT images. Phantom nodules of known sizes were used so that segmentation accuracy could be quantified in comparison to ground-truth volumes. Spherical nodules having 4.8, 9.5 and 16 mm diameters and 50 and 100 mg cc(-1) calcium contents were embedded in lung-tissue-simulating foam which was inserted in the thoracic cavity of a chest section phantom. CT scans of the phantom were acquired with a 16-slice scanner at various tube currents, pitches, fields-of-view and slice thicknesses. Scans were also taken using identical techniques either within the same day or five months apart for study of reproducibility. The phantom nodules were segmented with a three-dimensional active contour (3DAC) model that we previously developed for use on patient nodules. The percentage volume errors relative to the ground-truth volumes were estimated under the various imaging conditions. There was no statistically significant difference in volume error for repeated CT scans or scans taken with techniques where only pitch, field of view, or tube current (mA) were changed. However, the slice thickness significantly (p < 0.05) affected the volume error. Therefore, to evaluate nodule growth, consistent imaging conditions and high resolution should be used for acquisition of the serial CT scans, especially for smaller nodules. Understanding the effects of scanning and reconstruction parameters on volume measurements by 3DAC allows better interpretation of data and assessment of growth. Tracking nodule growth with computerized segmentation methods would reduce inter- and intraobserver variabilities.

The Lung Image Database Consortium (LIDC): a comparison of different size metrics for pulmonary nodule measurements

RATIONALE AND OBJECTIVES

The goal was to investigate the effects of choosing between different metrics in estimating the size of pulmonary nodules as a factor both of nodule characterization and of performance of computer aided detection systems, because the latter are always qualified with respect to a given size range of nodules.

MATERIALS AND METHODS

This study used 265 whole-lung CT scans documented by the Lung Image Database Consortium (LIDC) using their protocol for nodule evaluation. Each inspected lesion was reviewed independently by four experienced radiologists who provided boundary markings for nodules larger than 3 mm. Four size metrics, based on the boundary markings, were considered: a unidimensional and two bidimensional measures on a single image slice and a volumetric measurement based on all the image slices. The radiologist boundaries were processed and those with four markings were analyzed to characterize the interradiologist variation, while those with at least one marking were used to examine the difference between the metrics.

RESULTS

The processing of the annotations found 127 nodules marked by all of the four radiologists and an extended set of 518 nodules each having at least one observation with three-dimensional sizes ranging from 2.03 to 29.4 mm (average 7.05 mm, median 5.71 mm). A very high interobserver variation was observed for all these metrics: 95% of estimated standard deviations were in the following ranges for the three-dimensional, unidimensional, and two bidimensional size metrics, respectively (in mm): 0.49-1.25, 0.67-2.55, 0.78-2.11, and 0.96-2.69. Also, a very large difference among the metrics was observed: 0.95 probability-coverage region widths for the volume estimation conditional on unidimensional, and the two bidimensional size measurements of 10 mm were 7.32, 7.72, and 6.29 mm, respectively.

CONCLUSIONS

The selection of data subsets for performance evaluation is highly impacted by the size metric choice. The LIDC plans to include a single size measure for each nodule in its database. This metric is not intended as a gold standard for nodule size; rather, it is intended to facilitate the selection of unique repeatable size limited nodule subsets.

Lung cancer: interobserver agreement on interpretation of pulmonary findings at low-dose CT screening

PURPOSE

To evaluate agreement among radiologists on the interpretation of pulmonary findings at low-dose computed tomographic (CT) screening examinations for lung cancer.

MATERIALS AND METHODS

Institutional review board approval and informed consent were obtained. HIPAA guidelines were followed. Sixteen radiologists from the 10 National Lung Screening Trial screening centers of the National Cancer Institute's Lung Screening Study network reviewed image subsets from 135 baseline low-dose screening CT examinations in 135 trial participants (89 men, 46 women; mean age, 62.7 years +/- 5.4 [standard deviation]). Interpretations were classified into one of four of the following categories: noncalcified nodule 4 mm or larger in greatest transverse dimension (positive screening result); noncalcified nodule smaller than 4 mm in greatest transverse dimension (negative screening result); calcified, benign nodule (negative screening result); or no nodule (negative screening result). A recommendation for follow-up evaluation was obtained for each case. Interobserver agreement was evaluated by using the multirater kappa statistic and by using response frequencies and descriptive statistics.

RESULTS

Multirater kappa values ranged from 0.58 (for agreement among all four classifications; 95% confidence interval: 0.55, 0.61) to 0.64 (for agreement on classification as a positive or negative screening result; 95% confidence interval: 0.62, 0.65). The average percentage of reader pairs in agreement on the screening result per case (percentage agreement) was 82%. There was wide variation in the total number of abnormalities detected and classified as pulmonary nodules, with differences of up to more than twofold among radiologists. For cases classified as positive, multirater kappa for follow-up recommendations was 0.35.

CONCLUSION

Interobserver agreement was moderate to substantial; potential for considerable improvement exists. Clinical trial registration no. NCT00047385.

Lung lesion doubling times: values and variability based on method of volume determination

PURPOSE

To determine doubling times (DTs) of lung lesions based on volumetric measurements from thin-section CT imaging.

METHODS

Previously untreated patients with > or = two thin-section CT scans showing a focal lung lesion were identified. Lesion volumes were derived using direct volume measurements and volume calculations based on lesion area and diameter. Growth rates (GRs) were compared by tissue diagnosis and measurement technique.

RESULTS

54 lesions were evaluated including 8 benign lesions, 10 metastases, 3 lymphomas, 15 adenocarcinomas, 11 squamous carcinomas, and 7 miscellaneous lung cancers. Using direct volume measurements, median DTs were 453, 111, 15, 181, 139 and 137 days, respectively. Lung cancer DTs ranged from 23-2239 days. There were no significant differences in GRs among the different lesion types. There was considerable variability among GRs using different volume determination methods.

CONCLUSIONS

Lung cancer doubling times showed a substantial range, and different volume determination methods gave considerably different DTs.

Automated volumetry of solid pulmonary nodules in a phantom: accuracy across different CT scanner technologies

OBJECTIVES

The accuracy of automated volumetry for pulmonary nodules in a phantom using different CT scanner technologies from single-slice spiral CT (SSCT) to 64-slice multidetector-row CT (MDCT) was compared.

MATERIALS AND METHODS

A lung phantom with 5 different categories of pulmonary nodules was scanned using a single-slice spiral CT, a 4-slice MDCT, a 16-slice MDCT and a 64-slice MDCT. Each category comprised of 7-9 nodules each (total n = 40) with different known volumes. Standard dose and low dose protocols were performed using thin and thick collimation. Image data were reconstructed at the thinnest slice thickness. Data sets were analyzed with a dedicated volumetry software. Volumes of all nodules were calculated and compared.

RESULTS

Mean absolute percentage error (APE) for all nodules was 8.65% (+/-7.29%) for the SSCT, 10.26% (+/-8.25%) for the 4-slice MDCT, 8.19% (+/-7.57%) for the 16-slice MDCT and 7.89% (+/-7.39%) for the 64-slice MDCT. There was statistically significant influence of the scanner type, protocol, anatomic location, and nodule volume on APE, but overall, APEs were comparable.

CONCLUSION

Computer-aided volumetry showed accurate measurements in all tested scanner types. This finding has important implications for nodule assessment and follow-up.

The effect of lung volume on nodule size on CT

RATIONALE AND OBJECTIVES

We sought to determine how measures of nodule diameter and volume on computed tomography (CT) vary with changes in inspiratory level.

MATERIALS AND METHODS

CT scans were performed with inspiration suspended at total lung capacity (TLC) and then at residual volume (RV) in 41 subjects, in whom 75 indeterminate lung nodules were detected. A fully automated contouring program was used to segment the lungs; followed by segmentation of all nodules and the corresponding lobe using semiautomated contouring in both TLC and RV scans. The percent changes in lung and lobar volumes between TLC and RV were correlated with percent changes in nodule diameters and volumes.

RESULTS

Both nodule diameter and volume varied nonuniformly from TLC to RV-some nodules decreased in size, while others increased. There was a 16.8% mean change in absolute volume across all nodules. Stratified by size, the mean value of the absolute percent volume changes for nodules > or =5 mm and <5 mm were not significantly different (P = .26). Stratified by maximum attenuation, the mean value of the absolute percent volume changes between the TLC and RV series for noncalcified (17.7%, SD = 13.1) and completely calcified nodules (8.6% SD = 5.7) were significantly different (P < .05).

CONCLUSION

Significant differences in nodule size were measured between TLC and RV scans. This has important implications for standardizing acquisition protocols in any setting where size and, more important, size change are being used for purposes of lung cancer staging, nodule characterization, or treatment response assessment.

Accuracy of automated volumetry of pulmonary nodules across different multislice CT scanners

The purpose of this study was to compare the accuracy of an automated volumetry software for phantom pulmonary nodules across various 16-slice multislice spiral CT (MSCT) scanners from different vendors. A lung phantom containing five different nodule categories (intraparenchymal, around a vessel, vessel attached, pleural, and attached to the pleura), with each category comprised of 7-9 nodules (total, n = 40) of varying sizes (diameter 3-10 mm; volume 6.62 mm(3)-525 mm(3)), was scanned with four different 16-slice MSCT scanners (Siemens, GE, Philips, Toshiba). Routine and low-dose chest protocols with thin and thick collimations were applied. The data from all scanners were used for further analysis using a dedicated prototype volumetry software. Absolute percentage volume errors (APE) were calculated and compared. The mean APE for all nodules was 8.4% (+/-7.7%) for data acquired with the 16-slice Siemens scanner, 14.3% (+/-11.1%) for the GE scanner, 9.7% (+/-9.6%) for the Philips scanner and 7.5% (+/-7.2%) for the Toshiba scanner, respectively. The lowest APEs were found within the diameter size range of 5-10 mm and volumes >66 mm(3). Nodule volumetry is accurate with a reasonable volume error in data from different scanner vendors. This may have an important impact for intraindividual follow-up studies.

Ethical considerations in CT angiography

The rapid development and clinical deployment of CT angiography raises several important issues, including assurance of professional competence and technical quality, self-referral, the relative role of radiologists and cardiologists, appropriateness and proper indications, the detection and disposition of unexpected or incidental findings and the concern for the rapidly increasing costs of health care and imaging. These questions are properly addressed within the framework of medical ethics, including principles of beneficence, autonomy and justice.

The reliability of measuring physical characteristics of spiculated masses on mammography

The goal of this study was to assess the reliability of measurements of the physical characteristics of spiculated masses on mammography. The images used in this study were obtained from the Digital Database for Screening Mammography. Two experienced radiologists measured the properties of 21 images of spiculated masses. The length and width of all spicules and the major axis of the mass were measured. In addition, the observers counted the total number of spicules. Interobserver and intraobserver variability were evaluated using a hypothesis test for equivalence, the intraclass correlation coefficient (ICC) and Bland-Altman statistics. For an equivalence level of 30% of the mean of the senior radiologist's measurement, equivalence was achieved for the measurements of average spicule length (p<0.01), average spicule width (p = 0.03), the length of the major axis (p<0.01) and for the count of the number of spicules (p<0.01). Similarly, with the ICC analysis technique "excellent" inter-rater agreement was observed for the measurements of average spicule length (ICC = 0.770), the length of the major axis (ICC = 0.801) and for the count of the number of spicules (ICC = 0.780). "Fair to good" agreement was observed for the average spicule width (ICC = 0.561). Equivalence was also demonstrated for intraobserver measurements. Physical properties of spiculated masses can be measured reliably on mammography. The interobserver and intraobserver variability for this task is comparable with that reported for other measurements made on medical images.

Evaluation of lung MDCT nodule annotation across radiologists and methods

RATIONALE AND OBJECTIVES

Integral to the mission of the National Institutes of Health-sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary.

MATERIALS AND METHODS

The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists' spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects.

RESULTS

Across the combination of all nodules, volume and p-map model parameters were found to be significant at P < .05 for all methods, all radiologists, and all second-order interactions except one. The radiologist and methods variables accounted for 15% and 3.5% of the total p-map variance, respectively, and 40.4% and 31.1% of the total volume variance, respectively.

CONCLUSION

Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.

Response to chemotherapy in patients with lung metastases: how many nodules should be measured?

Objective: With the introduction of cross-sectional imaging methods the number of lesions per patient that can be evaluated is frequently large and most oncologists and study protocols use only one lesion or a few ‘representative’ lesions to evaluate chemotherapy response. Intra-patient response variability can therefore affect evaluation reproducibility. This study evaluates intra-individual variation in response to chemotherapy in patients with multiple lung metastases. Methods: We prospectively studied chest CT images of patients with solid tumors and pulmonary metastases under systemic chemotherapy being evaluated for tumor response. The response of 566 pulmonary nodules in 41 evaluations was determined by both WHO and RECIST criteria in order to determine intra-individual tumor response variation. Results: There was almost perfect agreement between the WHO and the RECIST criteria for the evaluation of tumor response. High intra-individual variability of tumor response was observed in a significant proportion of the evaluations. A new nodule was the main criterion for determination of disease progression. A mean of 35% of the total number of nodules of a patient have a response evaluation different from that calculated with all the nodules together. Conclusions: Intra-individual variation in tumor response of pulmonary metastases is elevated in some patients. Selecting any or some nodules for response evaluation could significantly influence therapeutic response perception.