Computerized comprehensive data analysis of lung imaging database consortium (LIDC)

Performance changes due to differences among annotating radiologists for training data in computerized lesion detection

PURPOSE

The quality and bias of annotations by annotators (e.g., radiologists) affect the performance changes in computer-aided detection (CAD) software using machine learning. We hypothesized that the difference in the years of experience in image interpretation among radiologists contributes to annotation variability. In this study, we focused on how the performance of CAD software changes with retraining by incorporating cases annotated by radiologists with varying experience.

METHODS

We used two types of CAD software for lung nodule detection in chest computed tomography images and cerebral aneurysm detection in magnetic resonance angiography images. Twelve radiologists with different years of experience independently annotated the lesions, and the performance changes were investigated by repeating the retraining of the CAD software twice, with the addition of cases annotated by each radiologist. Additionally, we investigated the effects of retraining using integrated annotations from multiple radiologists.

RESULTS

The performance of the CAD software after retraining differed among annotating radiologists. In some cases, the performance was degraded compared to that of the initial software. Retraining using integrated annotations showed different performance trends depending on the target CAD software, notably in cerebral aneurysm detection, where the performance decreased compared to using annotations from a single radiologist.

CONCLUSIONS

Although the performance of the CAD software after retraining varied among the annotating radiologists, no direct correlation with their experience was found. The performance trends differed according to the type of CAD software used when integrated annotations from multiple radiologists were used.

AAPM task group report 273: Recommendations on best practices for AI and machine learning for computer-aided diagnosis in medical imaging

Rapid advances in artificial intelligence (AI) and machine learning, and specifically in deep learning (DL) techniques, have enabled broad application of these methods in health care. The promise of the DL approach has spurred further interest in computer-aided diagnosis (CAD) development and applications using both "traditional" machine learning methods and newer DL-based methods. We use the term CAD-AI to refer to this expanded clinical decision support environment that uses traditional and DL-based AI methods. Numerous studies have been published to date on the development of machine learning tools for computer-aided, or AI-assisted, clinical tasks. However, most of these machine learning models are not ready for clinical deployment. It is of paramount importance to ensure that a clinical decision support tool undergoes proper training and rigorous validation of its generalizability and robustness before adoption for patient care in the clinic. To address these important issues, the American Association of Physicists in Medicine (AAPM) Computer-Aided Image Analysis Subcommittee (CADSC) is charged, in part, to develop recommendations on practices and standards for the development and performance assessment of computer-aided decision support systems. The committee has previously published two opinion papers on the evaluation of CAD systems and issues associated with user training and quality assurance of these systems in the clinic. With machine learning techniques continuing to evolve and CAD applications expanding to new stages of the patient care process, the current task group report considers the broader issues common to the development of most, if not all, CAD-AI applications and their translation from the bench to the clinic. The goal is to bring attention to the proper training and validation of machine learning algorithms that may improve their generalizability and reliability and accelerate the adoption of CAD-AI systems for clinical decision support.

Highly accurate model for prediction of lung nodule malignancy with CT scans

Computed tomography (CT) examinations are commonly used to predict lung nodule malignancy in patients, which are shown to improve noninvasive early diagnosis of lung cancer. It remains challenging for computational approaches to achieve performance comparable to experienced radiologists. Here we present NoduleX, a systematic approach to predict lung nodule malignancy from CT data, based on deep learning convolutional neural networks (CNN). For training and validation, we analyze >1000 lung nodules in images from the LIDC/IDRI cohort. All nodules were identified and classified by four experienced thoracic radiologists who participated in the LIDC project. NoduleX achieves high accuracy for nodule malignancy classification, with an AUC of ~0.99. This is commensurate with the analysis of the dataset by experienced radiologists. Our approach, NoduleX, provides an effective framework for highly accurate nodule malignancy prediction with the model trained on a large patient population. Our results are replicable with software available at http://bioinformatics.astate.edu/NoduleX .

Improvement in imaging diagnosis technique and modalities for solitary pulmonary nodules: from ground-glass opacity nodules to part-solid and solid nodules

With advances in CT technology and the popularity of low-dose CT as a device for lung cancer screening, the detection rate of sub-solid pulmonary nodules as well as solid nodules has been increased. Distinguishing solid from sub-solid features is an essential step in the CT evaluation of solitary pulmonary nodules (SPNs) because strategies for nodule characterization and guidelines for management are different for each category. In addition to conventional CT parameters, numerous novel concepts and modalities have been developed. Although there is currently no single effective method for differentiating malignant from benign nodules, growth rate measurement using volumetry, evaluation of tumor vascularity on dynamic helical CT, dual-energy CT and MRI and physiologic evaluation with PET/CT can all be useful for nodule characterization. New techniques such as tomosynthesis can improve detection over radiography alone. The purpose of this article is to enhance our understanding of the evidence-based strategies involved in diagnosing SPNs.

Seamless Insertion of Pulmonary Nodules in Chest CT Images

The availability of large medical image datasets is critical in many applications, such as training and testing of computer-aided diagnosis systems, evaluation of segmentation algorithms, and conducting perceptual studies. However, collection of data and establishment of ground truth for medical images are both costly and difficult. To address this problem, we are developing an image blending tool that allows users to modify or supplement existing datasets by seamlessly inserting a lesion extracted from a source image into a target image. In this study, we focus on the application of this tool to pulmonary nodules in chest CT exams. We minimize the impact of user skill on the perceived quality of the composite image by limiting user involvement to two simple steps: the user first draws a casual boundary around a nodule in the source, and, then, selects the center of desired insertion area in the target. We demonstrate the performance of our system on clinical samples, and report the results of a reader study evaluating the realism of inserted nodules compared to clinical nodules. We further evaluate our image blending techniques using phantoms simulated under different noise levels and reconstruction filters. Specifically, we compute the area under the ROC curve of the Hotelling observer (HO) and noise power spectrum of regions of interest enclosing native and inserted nodules, and compare the detectability, noise texture, and noise magnitude of inserted and native nodules. Our results indicate the viability of our approach for insertion of pulmonary nodules in clinical CT images.

Data analysis of the Lung Imaging Database Consortium and Image Database Resource Initiative

RATIONALE AND OBJECTIVES

The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) is the largest publicly available computed tomography (CT) image reference data set of lung nodules. In this article, a comprehensive data analysis of the data set and a uniform data model are presented with the purpose of facilitating potential researchers to have an in-depth understanding to and efficient use of the data set in their lung cancer-related investigations.

MATERIALS AND METHODS

A uniform data model was designed for representation and organization of various types of information contained in different source data files. A software tool was developed for the processing and analysis of the database, which 1) automatically aligns and graphically displays the nodule outlines marked manually by radiologists onto the corresponding CT images; 2) extracts diagnostic nodule characteristics annotated by radiologists; 3) calculates a variety of nodule image features based on the outlines of nodules, including diameter, volume, and degree of roundness, and so forth; 4) integrates all the extracted nodule information into the uniform data model and stores it in a common and easy-to-access data format; and 5) analyzes and summarizes various feature distributions of nodules in several different categories. Using this data processing and analysis tool, all 1018 CT scans from the data set were processed and analyzed for their statistical distribution.

RESULTS

The information contained in different source data files with different formats was extracted and integrated into a new and uniform data model. Based on the new data model, the statistical distributions of nodules in terms of nodule geometric features and diagnostic characteristics were summarized. In the LIDC/IDRI data set, 2655 nodules ≥3 mm, 5875 nodules <3 mm, and 7411 non-nodules are identified, respectively. Among the 2655 nodules, 1) 775, 488, 481, and 911 were marked by one, two, three, or four radiologists, respectively; 2) most of nodules ≥3 mm (85.7%) have a diameter <10.0 mm with the mean value of 6.72 mm; and 3) 10.87%, 31.4%, 38.8%, 16.4%, and 2.6% of nodules were assessed with a malignancy score of 1, 2, 3, 4, and 5, respectively.

CONCLUSIONS

This study demonstrates the usefulness of the proposed software tool to the potential users for an in-depth understanding of the LIDC/IDRI data set, therefore likely to be beneficial to their future investigations. The analysis results also demonstrate the distribution diversity of nodules characteristics, therefore being useful as a reference resource for assessing the performance of a new and existing nodule detection and/or segmentation schemes.

Computerized segmentation of pulmonary nodules depicted in CT examinations using freehand sketches

PURPOSE

To aid a consistent segmentation of pulmonary nodules, the authors describe a novel computerized scheme that utilizes a freehand sketching technique and an improved break-and-repair strategy.

METHODS

This developed scheme consists of two primary parts. The first part is freehand sketch analysis, where the freehand sketching not only serves a natural way of specifying the location of a nodule, but also provides a mechanism for inferring adaptive information (e.g., the mass center, the density, and the size) in regard to the nodule. The second part is an improved break-and-repair strategy. The improvement avoids the time-consuming ray-triangle intersections using spherical bins and replaces the original global implicit surface reconstruction with a local implicit surface fitting and blending scheme. The performance of this scheme, including accuracy and consistence, was assessed using 50 CT examinations in the Lung Image Database Consortium (LIDC). For each of these examinations, a single nodule was selected under the aid of a publically available tool to assure these nodules were diverse in size, location, and density. Two radiologists were asked to use the developed tool to segment these nodules twice at different times (at least three months apart). A Hausdorff distance based method was used to assess the discrepancies (agreements) between the computerized results and the results by the four radiologists in the LIDC as well as the inter- and intrareader agreements in freehand sketching.

RESULTS

The maximum and mean discrepancies in boundary outlines between the computerized scheme and the radiologists were 2.73 ± 1.32 mm and 1.01 ± 0.47 mm, respectively. When the nodules were classified (binned) into different size ranges, the maximum errors ranged from 1.91 to 4.13 mm; but smaller nodules had larger percentage discrepancies in term of size. Under the aid of the developed scheme, the inter- and intrareader variability in averaged maximum discrepancy across all types of pulmonary nodules were consistently smaller than 0.15 ± 0.07 mm. The computational cost in time of segmenting a pulmonary nodule ranged from 0.4 to 2.3 s with an average of 1.1 s for a typical desktop computer.

CONCLUSIONS

The experiments showed that this scheme could achieve a reasonable performance in nodule segmentation and demonstrated the merits of incorporating freehand sketching into pulmonary nodule segmentation.

Illustration of the obstacles in computerized lung segmentation using examples

PURPOSE

Automated lung volume segmentation is often a preprocessing step in quantitative lung computed tomography (CT) image analysis. The objective of this study is to identify the obstacles in computerized lung volume segmentation and illustrate those explicitly using real examples. Awareness of these "difficult" cases may be helpful for the development of a robust and consistent lung segmentation algorithm.

METHODS

We collected a large diverse dataset consisting of 2768 chest CT examinations acquired on 2292 subjects from various sources. These examinations cover a wide range of diseases, including lung cancer, chronic obstructive pulmonary disease, human immunodeficiency virus, pulmonary embolism, pneumonia, asthma, and interstitial lung disease (ILD). The CT acquisition protocols, including dose, scanners, and reconstruction kernels, vary significantly. After the application of a "neutral" thresholding-based approach to the collected CT examinations in a batch manner, the failed cases were subjectively identified and classified into different subgroups.

RESULTS

Totally, 121 failed examinations are identified, corresponding to a failure ratio of 4.4%. These failed cases are summarized as 11 different subgroups, which is further classified into 3 broad categories: (1) failure caused by diseases, (2) failure caused by anatomy variability, and (3) failure caused by external factors. The failure percentages in these categories are 62.0%, 32.2%, and 5.8%, respectively.

CONCLUSIONS

The presence of specific lung diseases (e.g., pulmonary nodules, ILD, and pneumonia) is the primary issue in computerized lung segmentation. The segmentation failures caused by external factors and anatomy variety are relatively low but unavoidable in practice. It is desirable to develop robust schemes to handle these issues in a single pass when a large number of CT examinations need to be analyzed.