Interactive computer-aided diagnosis of breast masses: computerized selection of visually similar image sets from a reference library

Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammograms

Deep convolutional neural networks (CNNs) have been widely used in various medical imaging tasks. However, due to the intrinsic locality of convolution operations, CNNs generally cannot model long-range dependencies well, which are important for accurately identifying or mapping corresponding breast lesion features computed from unregistered multiple mammograms. This motivated us to leverage the architecture of Multi-view Vision Transformers to capture long-range relationships of multiple mammograms from the same patient in one examination. For this purpose, we employed local transformer blocks to separately learn patch relationships within four mammograms acquired from two-view (CC/MLO) of two-side (right/left) breasts. The outputs from different views and sides were concatenated and fed into global transformer blocks, to jointly learn patch relationships between four images representing two different views of the left and right breasts. To evaluate the proposed model, we retrospectively assembled a dataset involving 949 sets of mammograms, which included 470 malignant cases and 479 normal or benign cases. We trained and evaluated the model using a five-fold cross-validation method. Without any arduous preprocessing steps (e.g., optimal window cropping, chest wall or pectoral muscle removal, two-view image registration, etc.), our four-image (two-view-two-side) transformer-based model achieves case classification performance with an area under ROC curve (AUC = 0.818 ± 0.039), which significantly outperforms AUC = 0.784 ± 0.016 achieved by the state-of-the-art multi-view CNNs (p = 0.009). It also outperforms two one-view-two-side models that achieve AUC of 0.724 ± 0.013 (CC view) and 0.769 ± 0.036 (MLO view), respectively. The study demonstrates the potential of using transformers to develop high-performing computer-aided diagnosis schemes that combine four mammograms.

Applying a random projection algorithm to optimize machine learning model for predicting peritoneal metastasis in gastric cancer patients using CT images

BACKGROUND AND OBJECTIVE

Non-invasively predicting the risk of cancer metastasis before surgery can play an essential role in determining which patients can benefit from neoadjuvant chemotherapy. This study aims to investigate and test the advantages of applying a random projection algorithm to develop and optimize a radiomics-based machine learning model to predict peritoneal metastasis in gastric cancer patients using a small and imbalanced computed tomography (CT) image dataset.

METHODS

A retrospective dataset involving CT images acquired from 159 patients is assembled, including 121 and 38 cases with and without peritoneal metastasis, respectively. A computer-aided detection scheme is first applied to segment primary gastric tumor volumes and initially compute 315 image features. Then, five gradients boosting machine (GBM) models embedded with five feature selection methods (including random projection algorithm, principal component analysis, least absolute shrinkage, and selection operator, maximum relevance and minimum redundancy, and recursive feature elimination) along with a synthetic minority oversampling technique, are built to predict the risk of peritoneal metastasis. All GBM models are trained and tested using a leave-one-case-out cross-validation method.

RESULTS

Results show that the GBM model embedded with a random projection algorithm yields a significantly higher prediction accuracy (71.2%) than the other four GBM models (p<0.05). The precision, sensitivity, and specificity of this optimal GBM model are 65.78%, 43.10%, and 87.12%, respectively.

CONCLUSIONS

This study demonstrates that CT images of the primary gastric tumors contain discriminatory information to predict the risk of peritoneal metastasis, and a random projection algorithm is a promising method to generate optimal feature vector, improving the performance of machine learning based prediction models.

An interactive system for computer-aided diagnosis of breast masses

Although mammography is the only clinically accepted imaging modality for screening the general population to detect breast cancer, interpreting mammograms is difficult with lower sensitivity and specificity. To provide radiologists "a visual aid" in interpreting mammograms, we developed and tested an interactive system for computer-aided detection and diagnosis (CAD) of mass-like cancers. Using this system, an observer can view CAD-cued mass regions depicted on one image and then query any suspicious regions (either cued or not cued by CAD). CAD scheme automatically segments the suspicious region or accepts manually defined region and computes a set of image features. Using content-based image retrieval (CBIR) algorithm, CAD searches for a set of reference images depicting "abnormalities" similar to the queried region. Based on image retrieval results and a decision algorithm, a classification score is assigned to the queried region. In this study, a reference database with 1,800 malignant mass regions and 1,800 benign and CAD-generated false-positive regions was used. A modified CBIR algorithm with a new function of stretching the attributes in the multi-dimensional space and decision scheme was optimized using a genetic algorithm. Using a leave-one-out testing method to classify suspicious mass regions, we compared the classification performance using two CBIR algorithms with either equally weighted or optimally stretched attributes. Using the modified CBIR algorithm, the area under receiver operating characteristic curve was significantly increased from 0.865 ± 0.006 to 0.897 ± 0.005 (p < 0.001). This study demonstrated the feasibility of developing an interactive CAD system with a large reference database and achieving improved performance.

Computer-aided detection of masses at mammography: interactive decision support versus prompts

PURPOSE

To compare effectiveness of an interactive computer-aided detection (CAD) system, in which CAD marks and their associated suspiciousness scores remain hidden unless their location is queried by the reader, with the effect of traditional CAD prompts used in current clinical practice for the detection of malignant masses on full-field digital mammograms.

MATERIALS AND METHODS

The requirement for institutional review board approval was waived for this retrospective observer study. Nine certified screening radiologists and three residents who were trained in breast imaging read 200 studies (63 studies containing at least one screen-detected mass, 17 false-negative studies, 20 false-positive studies, and 100 normal studies) twice, once with CAD prompts and once with interactive CAD. Localized findings were reported and scored by the readers. In the prompted mode, findings were recorded before and after activation of CAD. The partial area under the location receiver operating characteristic (ROC) curve for an interval of low false-positive fractions typical for screening, from 0 to 0.2, was computed for each reader and each mode. Differences in reader performance were analyzed by using software.

RESULTS

The average partial area under the location ROC curve with unaided reading was 0.57, and it increased to 0.62 with interactive CAD, while it remained unaffected by prompts. The difference in reader performance for unaided reading versus interactive CAD was statistically significant (P = .009).

CONCLUSION

When used as decision support, interactive use of CAD for malignant masses on mammograms may be more effective than the current use of CAD, which is aimed at the prevention of perceptual oversights.

Improving knowledge management through the support of image examination and data annotation using DICOM structured reporting

An important effort has been invested on improving the image diagnosis process in different medical areas using information technologies. The field of medical imaging involves two main data types: medical imaging and reports. Developments based on the DICOM standard have demonstrated to be a convenient and widespread solution among the medical community. The main objective of this work is to design a Web application prototype that will be able to improve diagnosis and follow-on of breast cancer patients. It is based on TRENCADIS middleware, which provides a knowledge-oriented storage model composed by federated repositories of DICOM image studies and DICOM-SR medical reports. The full structure and contents of the diagnosis reports are used as metadata for indexing images. The TRENCADIS infrastructure takes full advantage of Grid technologies by deploying multi-resource grid services that enable multiple views (reports schemes) of the knowledge database. The paper presents a real deployment of such Web application prototype in the Dr. Peset Hospital providing radiologists with a tool to create, store and search diagnostic reports based on breast cancer explorations (mammography, magnetic resonance, ultrasound, pre-surgery biopsy and post-surgery biopsy), improving support for diagnostics decisions. A technical details for use cases (outlining enhanced multi-resource grid services communication and processing steps) and interactions between actors and the deployed prototype are described. As a result, information is more structured, the logic is clearer, network messages have been reduced and, in general, the system is more resistant to failures.

Clinically missed cancer: how effectively can radiologists use computer-aided detection?

OBJECTIVE

The purpose of this study was to determine the effectiveness with which radiologists can use computer-aided detection (CADe) to detect cancer missed at screening.

MATERIALS AND METHODS

An observer study was performed to measure the ability of radiologists to detect breast cancer on mammograms with and without CADe. The images in the study were from 300 analog mammographic examinations. In 234 cases the mammograms were read clinically as normal and free of cancer for at least 2 subsequent years. In the other 66 cases, cancers were missed clinically. In 256 cases, current and previous mammograms were available. Eight radiologists read the dataset and recorded a BI-RADS assessment, the location of the lesion, and their level of confidence that the patient should be recalled for diagnostic workup for each suspicious lesion. Jackknife alternative free-response receiver operating characteristic analysis was used.

RESULTS

The jackknife alternative free-response receiver operating characteristic figure of merit was 0.641 without aid and 0.659 with aid (p = 0.06; 95% CI, -0.001 to 0.036). The sensitivity increased 9.9% (95% CI, 3.4-19%) and the callback rate 12.1% (95% CI, 7.3-20%) with CADe. Both increases were statistically significant (p < 0.001). Radiologists on average ignored 71% of correct computer prompts.

CONCLUSION

Use of CADe can increase radiologist sensitivity 10% with a comparable increase in recall rate. There is potential for CADe to have a bigger clinical impact because radiologists failed to recognize a correct computer prompt in 71% of missed cancer cases [corrected].

Improving the performance of computer-aided detection of subtle breast masses using an adaptive cueing method

Current computer-aided detection (CAD) schemes for detecting mammographic masses have several limitations including high correlation with radiologists' detection and cueing most subtle masses only on one view. To increase CAD sensitivity in cueing more subtle masses that are likely missed and/or overlooked by radiologists without increasing false-positive rates, we investigated a new case-dependent cueing method by combining the original CAD-generated detection scores with a computed bilateral mammographic density asymmetry index. Using the new method, we adaptively raise the CAD-generated scores of the regions detected on 'high-risk' cases to cue more subtle mass regions and reduce the CAD scores of the regions detected on 'low-risk' cases to discard more false-positive regions. A testing dataset involving 78 positive and 338 negative cases was used to test this adaptive cueing method. Each positive case involves two sequential examinations in which the mass was detected in 'current' examination and missed in 'prior' examination but detected in a retrospective review by radiologists. Applying to this dataset, a pre-optimized CAD scheme yielded 75% case-based and 55% region-based sensitivity on 'current' examinations at a false-positive rate of 0.25 per image. CAD sensitivity was reduced to 42% (case based) and 27% (region based) on 'prior' examinations. Using the new cueing method, case-based and region-based sensitivity could maximally increase 9% and 33% on the 'prior' examinations, respectively. The percentages of the masses cued on two views also increased from 27% to 65%. The study demonstrated that using this adaptive cueing method enabled us to help CAD cue more subtle cancers without increasing the false-positive cueing rate.

Fissures segmentation using surface features: content-based retrieval for mammographic mass using ensemble classifier

RATIONALE AND OBJECTIVES

Accurate classification is critical in mammography computer-aided diagnosis using content-based image retrieval approaches (CBIR CAD). The objectives of this study were to: 1) develop an accurate ensemble classifier based on domain knowledge and a robust feature selection method for CBIR CAD; 2) propose three new features; and 3) assess the performance of the proposed method and new features by using a relatively large imaging data set.

MATERIALS AND METHODS

The data set used in this study consisted of 2114 regions of interest (ROI) extracted from a publicly available image database. The proposed ensemble classifier method we called E-DGA-KNN included four steps. In the first step, 804 ROIs depict masses were divided into five classes according to their boundary types. Then, each class of ROI with an equal number of negative ROIs were put together to create a sub-database. Second, a dual-stage genetic algorithm, which was called DGA, was applied on those five sub-databases for feature selection and weights determination respectively. In the third step, five base K-nearest neighbor (KNN) classifiers were created by using the results of the second step on 2114 ROIs, and five detection scores for a given queried ROI were obtained. Finally, these classifiers are combined to yield a final classification. The performances of the proposed methods were evaluated by using receiver operating characteristic (ROC) analysis. A comparison with eight different methods on the data set was provided which include the stepwise linear discriminative analysis algorithm (SLDA) and particle swarm optimization (PSO) algorithm with KNN classifier.

RESULTS

When four hybrid feature selection methods were applied with single KNN classifier (ie, DGA-KNN, SLDA-WGA-KNN, SLDA-PSO-KNN, GA-PSO-KNN) and the proposed E-DGA-KNN method to the data set, the computed areas under the ROC curve (Az) were 0.8782 ± 0.0080, 0.8675 ± 0.0081, 0.8623 ± 0.0083, 0.8725 ± 0.0079, and 0.8927 ± 0.0073, respectively. If all features and single KNN classifier were used, the Az value was 0.8478 ± 0.0088. Az values were 0.8592 ± 0.0083 and 0.8632 ± 0.0081 when SLDA or GA algorithm used alone.

CONCLUSIONS

In this study, an ensemble classifier based on domain knowledge and a dual-stage feature selection method was proposed. Evaluation results indicated that the proposed method achieved largest value of ROC compared to other algorithms. The proposed method shows better performance and has the potential to improve the performance of CBIR CAD in interpreting and analyzing mammograms.

Assessment of performance and reliability of computer-aided detection scheme using content-based image retrieval approach and limited reference database

Content-based image retrieval approach was used in our computer-aided detection (CAD) schemes for breast cancer detection with mammography. In this study, we assessed CAD performance and reliability using a reference database including 1500 positive (breast mass) regions of interest (ROIs) and 1500 normal ROIs. To test the relationship between CAD performance and the similarity level between the queried ROI and the retrieved ROIs, we applied a set of similarity thresholds to the retrieved similar ROIs selected by the CAD schemes for all queried suspicious regions, and used only the ROIs that were above the threshold for assessing CAD performance at each threshold level. Using the leave-one-out testing method, we computed areas under receiver operating characteristic (ROC) curves (A(Z)) to assess CAD performance. The experimental results showed that as threshold increase, (1) less true positive ROIs can be referenced in the database than normal ROIs and (2) the A(Z) value was monotonically increased from 0.854 ± 0.004 to 0.932 ± 0.016. This study suggests that (1) in order to more accurately detect and diagnose subtle masses, a large and diverse database is required, and (2) assessing the reliability of the decision scores based on the similarity measurement is important in application of the CBIR-based CAD schemes when the limited database is used.

A boosting framework for visuality-preserving distance metric learning and its application to medical image retrieval

Similarity measurement is a critical component in content-based image retrieval systems, and learning a good distance metric can significantly improve retrieval performance. However, despite extensive study, there are several major shortcomings with the existing approaches for distance metric learning that can significantly affect their application to medical image retrieval. In particular, "similarity" can mean very different things in image retrieval: resemblance in visual appearance (e.g., two images that look like one another) or similarity in semantic annotation (e.g., two images of tumors that look quite different yet are both malignant). Current approaches for distance metric learning typically address only one goal without consideration of the other. This is problematic for medical image retrieval where the goal is to assist doctors in decision making. In these applications, given a query image, the goal is to retrieve similar images from a reference library whose semantic annotations could provide the medical professional with greater insight into the possible interpretations of the query image. If the system were to retrieve images that did not look like the query, then users would be less likely to trust the system; on the other hand, retrieving images that appear superficially similar to the query but are semantically unrelated is undesirable because that could lead users toward an incorrect diagnosis. Hence, learning a distance metric that preserves both visual resemblance and semantic similarity is important. We emphasize that, although our study is focused on medical image retrieval, the problem addressed in this work is critical to many image retrieval systems. We present a boosting framework for distance metric learning that aims to preserve both visual and semantic similarities. The boosting framework first learns a binary representation using side information, in the form of labeled pairs, and then computes the distance as a weighted Hamming distance using the learned binary representation. A boosting algorithm is presented to efficiently learn the distance function. We evaluate the proposed algorithm on a mammographic image reference library with an Interactive Search-Assisted Decision Support (ISADS) system and on the medical image data set from ImageCLEF. Our results show that the boosting framework compares favorably to state-of-the-art approaches for distance metric learning in retrieval accuracy, with much lower computational cost. Additional evaluation with the COREL collection shows that our algorithm works well for regular image data sets.

Computer-Aided Diagnosis in Mammography Using Content-based Image Retrieval Approaches: Current Status and Future Perspectives

As the rapid advance of digital imaging technologies, the content-based image retrieval (CBIR) has became one of the most vivid research areas in computer vision. In the last several years, developing computer-aided detection and/or diagnosis (CAD) schemes that use CBIR to search for the clinically relevant and visually similar medical images (or regions) depicting suspicious lesions has also been attracting research interest. CBIR-based CAD schemes have potential to provide radiologists with "visual aid" and increase their confidence in accepting CAD-cued results in the decision making. The CAD performance and reliability depends on a number of factors including the optimization of lesion segmentation, feature selection, reference database size, computational efficiency, and relationship between the clinical relevance and visual similarity of the CAD results. By presenting and comparing a number of approaches commonly used in previous studies, this article identifies and discusses the optimal approaches in developing CBIR-based CAD schemes and assessing their performance. Although preliminary studies have suggested that using CBIR-based CAD schemes might improve radiologists' performance and/or increase their confidence in the decision making, this technology is still in the early development stage. Much research work is needed before the CBIR-based CAD schemes can be accepted in the clinical practice.

Improving performance of computer-aided detection scheme by combining results from two machine learning classifiers

RATIONALE AND OBJECTIVES

Global data-based and local instance-based machine-learning methods and classifiers have been widely used to optimize computer-aided detection and diagnosis (CAD) schemes to classify between true-positive and false-positive detections. In this study, the correlation between these two types of classifiers was investigated using a new independent testing data set, and the potential improvement of a CAD scheme's performance by combining the results of the two classifiers in detecting breast masses was assessed.

MATERIALS AND METHODS

The CAD scheme first used image filtering and a multilayer topographic region growth algorithm to detect and segment suspicious mass regions. The scheme then used an image feature-based classifier to classify these regions into true-positive and false-positive regions. Two classifiers were used in this study. One was a global data-based machine-learning classifier, an artificial neural network (ANN), and the other was a local instance-based machine-learning classifier, a k-nearest neighbor (KNN) algorithm. An independent image database including 400 mammographic examinations was used in this study. Of these, 200 were cancer cases and 200 were negative cases. The preoptimized CAD scheme was applied twice to the database using the two different classifiers. The correlation between the two sets of classification results was analyzed. Three sets of CAD performance results using the ANN, KNN, and average detection scores from both classifiers were assessed and compared using the free-response receiver-operating characteristic method.

RESULTS

The results showed that the ANN achieved higher performance than the KNN algorithm, with a normalized area under the performance curve (AUC) of 0.891 versus 0.845. The correlation coefficients between the detection scores generated by the two classifiers were 0.436 and 0.161 for the true-positive and false-positive detections, respectively. The average detection scores of the two classifiers improved CAD performance and reliability by increasing the AUC to 0.912 and reducing the standard error of the estimated AUC by 14.4%. The detection sensitivity was also increased from 75.8% (ANN) and 65.9% (KNN) to 80.3% at a false-positive detection rate of 0.3 per image.

CONCLUSIONS

This study demonstrates that two global data-based and local data-based machine-learning classifiers (ANN and KNN) generated low correlated detection results and that combining the detection scores of these two classifiers significantly improved overall CAD performance (P < .01) and reduced standard error in CAD performance assessment.

Improving performance of content-based image retrieval schemes in searching for similar breast mass regions: an assessment

This study aims to assess three methods commonly used in content-based image retrieval (CBIR) schemes and investigate the approaches to improve scheme performance. A reference database involving 3000 regions of interest (ROIs) was established. Among them, 400 ROIs were randomly selected to form a testing dataset. Three methods, namely mutual information, Pearson's correlation and a multi-feature-based k-nearest neighbor (KNN) algorithm, were applied to search for the 15 'the most similar' reference ROIs to each testing ROI. The clinical relevance and visual similarity of searching results were evaluated using the areas under receiver operating characteristic (ROC) curves (A(Z)) and average mean square difference (MSD) of the mass boundary spiculation level ratings between testing and selected ROIs, respectively. The results showed that the A(Z) values were 0.893 +/- 0.009, 0.606 +/- 0.021 and 0.699 +/- 0.026 for the use of KNN, mutual information and Pearson's correlation, respectively. The A(Z) values increased to 0.724 +/- 0.017 and 0.787 +/- 0.016 for mutual information and Pearson's correlation when using ROIs with the size adaptively adjusted based on actual mass size. The corresponding MSD values were 2.107 +/- 0.718, 2.301 +/- 0.733 and 2.298 +/- 0.743. The study demonstrates that due to the diversity of medical images, CBIR schemes using multiple image features and mass size-based ROIs can achieve significantly improved performance.

Mammographic images segmentation using texture descriptors

Tissue classification in mammography can help the diagnosis of breast cancer by separating healthy tissue from lesions. We present herein the use of three texture descriptors for breast tissue segmentation purposes: the Sum Histogram, the Gray Level Co-Occurrence Matrix (GLCM) and the Local Binary Pattern (LBP). A modification of the LBP is also proposed for a better distinction of the tissues. In order to segment the image into its tissues, these descriptors are compared using a fidelity index and two clustering algorithms: k-Means and SOM (Self-Organizing Maps).

Automated detection of breast mass spiculation levels and evaluation of scheme performance

RATIONALE AND OBJECTIVES

Although the spiculation levels of breast mass boundaries are a primary sign of malignancy for masses detected on mammography, developing an automated computerized method to detect spiculation levels and quantitatively evaluating the performance of such a method is a difficult task. The objectives of this study were to (1) develop and test a new method to improve mass segmentation and detect mass boundary spiculation levels and (2) assess the performance of this method using a relatively large imaging data set.

MATERIALS AND METHODS

The fully automated method developed for this study includes three image-processing steps. In the first step, the principle of maximum entropy is applied in the selected region of interest (ROI) after correcting the background trend to enhance the initial outlines of a mass. In the second step, an active-contour model is used to refine the initial outlines. In the third step, spiculated lines connected to the mass boundary are detected and identified using a special line detector. A quantitative spiculation index is computed to assess the degree of spiculation. To develop and evaluate this automated method, 211 ROIs depicting masses were extracted from a publicly available image database. Among these ROIs, 106 depicted circumscribed mass regions and 105 involved spiculated mass regions. The performance of the method was evaluated using receiver-operating characteristic (ROC) analysis.

RESULTS

The computed area under the ROC curve, when applying the method to the data set, was 0.701 +/- 0.027. By setting up a threshold at a spiculation index of 5.0, the method achieved an overall classification accuracy of 66.4%, with 54.3% sensitivity and 78.3% specificity.

CONCLUSIONS

In this study, a new computerized method with a number of unique characteristics was developed to detect spiculated mass regions, and a simple spiculation index was applied to quantify mass spiculation levels. Although this quantitative index can be used to distinguish between spiculated and circumscribed masses, the results also suggest that the automated detection of mass spiculation levels remains a technical challenge.

Optimization of reference library used in content-based medical image retrieval scheme

Building an optimal image reference library is a critical step in developing the interactive computer-aided detection and diagnosis (I-CAD) systems of medical images using content-based image retrieval (CBIR) schemes. In this study, the authors conducted two experiments to investigate (1) the relationship between I-CAD performance and size of reference library and (2) a new reference selection strategy to optimize the library and improve I-CAD performance. The authors assembled a reference library that includes 3153 regions of interest (ROI) depicting either malignant masses (1592) or CAD-cued false-positive regions (1561) and an independent testing data set including 200 masses and 200 false-positive regions. A CBIR scheme using a distance-weighted K-nearest neighbor algorithm is applied to retrieve references that are considered similar to the testing sample from the library. The area under receiver operating characteristic curve (Az) is used as an index to evaluate the I-CAD performance. In the first experiment, the authors systematically increased reference library size and tested I-CAD performance. The result indicates that scheme performance improves initially from Az= 0.715 to 0.874 and then plateaus when the library size reaches approximately half of its maximum capacity. In the second experiment, based on the hypothesis that a ROI should be removed if it performs poorly compared to a group of similar ROIs in a large and diverse reference library, the authors applied a new strategy to identify "poorly effective" references. By removing 174 identified ROIs from the reference library, I-CAD performance significantly increases to Az = 0.914 (p < 0.01). The study demonstrates that increasing reference library size and removing poorly effective references can significantly improve I-CAD performance.