Recent advances in observer performance methodology: jackknife free-response ROC (JAFROC)

Part I: prostate cancer detection, artificial intelligence for prostate cancer and how we measure diagnostic performance: a comprehensive review

MRI has firmly established itself as a mainstay for the detection, staging and surveillance of prostate cancer. Despite its success, prostate MRI continues to suffer from poor inter-reader variability and a low positive predictive value. The recent emergence of Artificial Intelligence (AI) to potentially improve diagnostic performance shows great potential. Understanding and interpreting the AI landscape as well as ever-increasing research literature, however, is difficult. This is in part due to widely varying study design and reporting techniques. This paper aims to address this need by first outlining the different types of AI used for the detection and diagnosis of prostate cancer, next deciphering how data collection methods, statistical analysis metrics (such as ROC and FROC analysis) and end points/outcomes (lesion detection vs. case diagnosis) affect the performance and limit the ability to compare between studies. Finally, this work explores the need for appropriately enriched investigational datasets and proper ground truth, and provides guidance on how to best conduct AI prostate MRI studies. Published in parallel, a clinical study applying this suggested study design was applied to review and report a multiple-reader multiple-case clinical study of 150 bi-parametric prostate MRI studies across nine readers, measuring physician performance both with and without the use of a recently FDA cleared Artificial Intelligence software.1.

Compressed sensing with deep learning reconstruction: Improving capability of gadolinium-EOB-enhanced 3D T1WI

PURPOSE

The purpose of this study was to determine the utility of compressed sensing (CS) with deep learning reconstruction (DLR) for improving spatial resolution, image quality and focal liver lesion detection on high-resolution contrast-enhanced T1-weighted imaging (HR-CE-T1WI) obtained by CS with DLR as compared with conventional CE-T1WI with parallel imaging (PI).

METHODS

Seventy-seven participants with focal liver lesions underwent conventional CE-T1WI with PI and HR-CE-T1WI, surgical resection, transarterial chemoembolization, and radiofrequency ablation, followed by histopathological or >2-year follow-up examinations in our hospital. Signal-to-noise ratios (SNRs) of liver, spleen and kidney were calculated for each patient, after which each SNR was compared by means of paired t-test. To compare focal lesion detection capabilities of the two methods, a 5-point visual scoring system was adopted for a per lesion basis analysis. Jackknife free-response receiver operating characteristic (JAFROC) analysis was then performed, while sensitivity and false positive rates (/data set) for consensus assessment of the two methods were also compared by using McNemar's test or the signed rank test.

RESULTS

Each SNR of HR-CE-T1WI was significantly higher than that of conventional CE-T1WI with PI (p < 0.05). Sensitivities for consensus assessment showed that HR-CE-MRI had significantly higher sensitivity than conventional CE-T1WI with PI (p = 0.004). Moreover, there were significantly fewer FP/cases for HR-CE-T1WI than for conventional CE-T1WI with PI (p = 0.04).

CONCLUSION

CS with DLR are useful for improving spatial resolution, image quality and focal liver lesion detection capability of Gd-EOB-DTPA enhanced 3D T1WI without any need for longer breath-holding time.

Surgical and Radiology Trainees' Proficiency in Reading Mammograms: the Importance of Education for Cancer Localisation

Medical imaging with mammography plays a very important role in screening and diagnosis of breast cancer, Australia's most common female cancer. The visualisation of cancers on mammograms often forms a diagnosis and guidance for radiologists and breast surgeons, and education platforms that provide real cases in a simulated testing environment have been shown to improve observer performance for radiologists. This study reports on the performance of surgical and radiology trainees in locating breast cancers. An enriched test set of 20 mammography cases (6 cancer and 14 cancer free) was created, and 18 surgical trainees and 32 radiology trainees reviewed the cases via the Breast Screen Reader Assessment Strategy (BREAST) platform and marked any lesions identifiable. Further analysis of performance with high- and low-density cases was undertaken, and standard metrics including sensitivity and specificity. Radiology trainees performed significantly better than surgical trainees in terms of specificity (0.72 vs. 0.35; P < 0.01). No significant differences were observed between the surgical and radiology trainees in sensitivity or lesion sensitivity. Mixed results were obtained with participants regarding breast density, with higher density cases generally having lower performance. The higher specificity of the radiology trainees compared to the surgical trainees likely represents less exposure to negative mammography cases. The use of high-fidelity simulated self-test environments like BREAST is able to benchmark, understand and build strategies for improving cancer education in a safe environment, including identifying challenging scenarios like breast density for enhanced training.

The diagnostic performance and clinical value of deep learning-based nodule detection system concerning influence of location of pulmonary nodule

BACKGROUND

The deep learning-based nodule detection (DLD) system improves nodule detection performance of observers on chest radiographs (CXRs). However, its performance in different pulmonary nodule (PN) locations remains unknown.

METHODS

We divided the CXR intrathoracic region into non-danger zone (NDZ) and danger zone (DZ). The DZ included the lung apices, paramediastinal areas, and retrodiaphragmatic areas, where nodules could be missed. We used a dataset of 300 CXRs (100 normal and 200 abnormal images with 216 PNs [107 NDZ and 109 DZ nodules]). Eight observers (two thoracic radiologists [TRs], two non-thoracic radiologists [NTRs], and four radiology residents [RRs]) interpreted each radiograph with and without the DLD system. The metric of lesion localization fraction (LLF; the number of correctly localized lesions divided by the total number of true lesions) was used to evaluate the diagnostic performance according to the nodule location.

RESULTS

The DLD system demonstrated a lower LLF for the detection of DZ nodules (64.2) than that of NDZ nodules (83.2, p = 0.008). For DZ nodule detection, the LLF of the DLD system (64.2) was lower than that of TRs (81.7, p < 0.001), which was comparable to that of NTRs (56.4, p = 0.531) and RRs (56.7, p = 0.459). Nonetheless, the LLF of RRs significantly improved from 56.7 to 65.6 using the DLD system (p = 0.021) for DZ nodule detection.

CONCLUSION

The performance of the DLD system was lower in the detection of DZ nodules compared to that of NDZ nodules. Nonetheless, RR performance in detecting DZ nodules improved upon using the DLD system.

CRITICAL RELEVANCE STATEMENT

Despite the deep learning-based nodule detection system's limitations in detecting danger zone nodules, it proves beneficial for less-experienced observers by providing valuable assistance in identifying these nodules, thereby advancing nodule detection in clinical practice.

KEY POINTS

• The deep learning-based nodule detection (DLD) system can improve the diagnostic performance of observers in nodule detection. • The DLD system shows poor diagnostic performance in detecting danger zone nodules. • For less-experienced observers, the DLD system is helpful in detecting danger zone nodules.

Computed Tomography of the Head : A Systematic Review on Acquisition and Reconstruction Techniques to Reduce Radiation Dose

In 1971, the first computed tomography (CT) scan was performed on a patient's brain. Clinical CT systems were introduced in 1974 and dedicated to head imaging only. New technological developments, broader availability, and the clinical success of CT led to a steady growth in examination numbers. Most frequent indications for non-contrast CT (NCCT) of the head include the assessment of ischemia and stroke, intracranial hemorrhage and trauma, while CT angiography (CTA) has become the standard for first-line cerebrovascular evaluation; however, resulting improvements in patient management and clinical outcomes come at the cost of radiation exposure, increasing the risk for secondary morbidity. Therefore, radiation dose optimization should always be part of technical advancements in CT imaging but how can the dose be optimized? What dose reduction can be achieved without compromising diagnostic value, and what is the potential of the upcoming technologies artificial intelligence and photon counting CT? In this article, we look for answers to these questions by reviewing dose reduction techniques with respect to the major clinical indications of NCCT and CTA of the head, including a brief perspective on what to expect from current and future developments in CT technology with respect to radiation dose optimization.

Multiclass Segmentation of Breast Tissue and Suspicious Findings: A Simulation-Based Study for the Development of Self-Steering Tomosynthesis

In breast tomosynthesis, multiple low-dose projections are acquired in a single scanning direction over a limited angular range to produce cross-sectional planes through the breast for three-dimensional imaging interpretation. We built a next-generation tomosynthesis system capable of multidirectional source motion with the intent to customize scanning motions around "suspicious findings". Customized acquisitions can improve the image quality in areas that require increased scrutiny, such as breast cancers, architectural distortions, and dense clusters. In this paper, virtual clinical trial techniques were used to analyze whether a finding or area at high risk of masking cancers can be detected in a single low-dose projection and thus be used for motion planning. This represents a step towards customizing the subsequent low-dose projection acquisitions autonomously, guided by the first low-dose projection; we call this technique "self-steering tomosynthesis." A U-Net was used to classify the low-dose projections into "risk classes" in simulated breasts with soft-tissue lesions; class probabilities were modified using post hoc Dirichlet calibration (DC). DC improved the multiclass segmentation (Dice = 0.43 vs. 0.28 before DC) and significantly reduced false positives (FPs) from the class of the highest risk of masking (sensitivity = 81.3% at 2 FPs per image vs. 76.0%). This simulation-based study demonstrated the feasibility of identifying suspicious areas using a single low-dose projection for self-steering tomosynthesis.

Comparison of two-view versus single-view digital breast tomosynthesis and 2D-mammography in breast cancer surveillance imaging

PURPOSE

Limited work has been performed for the implementation of digital breast tomosynthesis (DBT) in breast cancer surveillance imaging. The aim of this study was to investigate the differences between two different DBT implementations in breast cancer surveillance imaging, for patients with a personal history of breast cancer.

METHOD

The DBT implementations investigated were: (1) 2-view 2D digital mammography and 2-view DBT (2vDM&2vDBT) (2) 1-view (cranial-caudal) DM and 1-view (mediolateral-oblique) DBT (1vDM&1vDBT). Clinical performance of these two implementations was assessed retrospectively using observer studies with 118 sets of real patient images, from a single imaging centre, and six observers. Sensitivity, specificity and area under the curve (AUC) using the Jack-knife alternative free-response receiver operating characteristics (JAFROC) analysis were evaluated.

RESULTS

Results suggest that the two DBT implementations are not significantly different in terms of sensitivity, specificity and AUC. When looking at the two main different lesion types, non-calcifications and calcifications, and two different density levels, no difference in the performance of the two DBT implementations was found.

CONCLUSIONS

Since 1vDM&1vDBT exposes the patient to half the dose of 2vDM&2vDBT, it might be worth considering 1vDM&1vDBT in breast cancer surveillance imaging. However, larger studies are required to conclude on this matter.

Deep learning applied to automatic disease detection using chest X-rays

Deep learning (DL) has shown rapid advancement and considerable promise when applied to the automatic detection of diseases using CXRs. This is important given the widespread use of CXRs across the world in diagnosing significant pathologies, and the lack of trained radiologists to report them. This review article introduces the basic concepts of DL as applied to CXR image analysis including basic deep neural network (DNN) structure, the use of transfer learning and the application of data augmentation. It then reviews the current literature on how DNN models have been applied to the detection of common CXR abnormalities (e.g. lung nodules, pneumonia, tuberculosis and pneumothorax) over the last few years. This includes DL approaches employed for the classification of multiple different diseases (multi-class classification). Performance of different techniques and models and their comparison with human observers are presented. Some of the challenges facing DNN models, including their future implementation and relationships to radiologists, are also discussed.

Digital mammographic interpretation by UK radiographer mammographers: A JAFROC analysis of observer performance

INTRODUCTION

Radiologists utilise mammography test sets to bench mark their performance against recognised standards. Using a validated test set, this study compares the performance of radiographer readers against previous test results for radiologists.

METHODS

Under similar test conditions radiographer readers were given an established test set of 60 mammograms and tasked to identify breast cancer, they were measured against their ability to identify, locate and give a confidence level for cancer being present on a standard set of mammographic images. The results were then compared to previously published results for radiologists for similar or the same test sets.

RESULTS

The 10 radiographer readers demonstrated similar results to radiologists and for lesion sensitivity were the highest scoring group. The study group score a sensitivity of 83; a specificity of 69.3 and lesion sensitivity of 74.8 with ROC and JAFROC scores of 0.86 and 0.74 respectively.

CONCLUSION

Under test conditions radiographers are able to identify and accurately locate breast cancer in a range of complex mammographic backgrounds.

IMPLICATIONS FOR PRACTICE

The study was performed under experimental conditions with results comparable to breast radiologists under similar conditions, translation of these findings into clinical practice will help address access and capacity issues in the timely identification and diagnosis of breast cancer.

Understanding diagnostic test accuracy studies and systematic reviews: A primer for medical radiation technologists

Diagnostic test accuracy studies are performed in order to determine the value of a diagnostic test. Primary studies of diagnostic accuracy describe the accuracy of a test calculated using the number of true and false positive and negative cases of a given diagnosis of interest. Systematic reviews of diagnostic test accuracy are performed to identify accuracy of a test but also to investigate where the test might sit in a diagnostic pathway or determine how different tests compare against each other. This introductory discussion paper aims to give the reader an overview of the various features in primary and secondary diagnostic test accuracy study designs for medical imaging professionals.

Reading High Breast Density Mammograms: Differences in Diagnostic Performance between Radiologists from Hong Kong SAR/Guangdong Province in China and Australia

Background:

Variations in the performance of radiologists reading mammographic images are well reported, but key parameters explaining such variations in different countries are not fully explored. The main aim of this study is to investigate performances of Chinese (Hong Kong SAR and Guangdong Province) and Australian radiologists in interpreting dense breast mammographic images.

Methods:

A test set, contained 60 mammographic examinations with high breast density, was used to assess radiologists’ performance. Twelve Chinese and thirteen Australian radiologists read all the cases independently and were asked to identify all lesions and provide a grade from 1 to 5 to each lesion. Case sensitivity, specificity, lesion sensitivity, AUC and JAFROC were used to assess radiologists’ performances. Demographic information and reading experience were also collected from the readers. Performance scores were compared between the two populations and the relationships between performance scores and their reading experience were discovered.

Results:

For radiologists who were less than 40-year-old, lesion sensitivity, AUC and JAFROC were significantly lower in Chinese radiologists than those in Australian (52.10% vs 71.45%, p=0.043; 0.76 vs 0.84, p=0.031; 0.59 vs 0.72, p=0.045; respectively). Australian radiologists with less than 10 years of reading experience had higher AUC and JAFROC scores compared with their Chinese counterparts (0.83 vs 0.76, p=0.039; 0.70 vs 0.56, p=0.020, respectively).

Conclusions:

We found that younger Australian radiologists performed better at reading dense breast cases which is likely to be linked to intensive fellowship training, immersion in a screening program and exposure to the benefits of a performance-measuring education tool.

A Protease-Activated Fluorescent Probe Allows Rapid Visualization of Keratinocyte Carcinoma during Excision

Keratinocyte carcinomas, including basal and squamous cell carcinomas, are the most common human cancers worldwide. While 75% of all keratinocyte carcinoma (4 million annual cases in the United States) are treated with conventional excision, this surgical modality has much lower cure rates than Mohs micrographic surgery, likely due to the bread-loaf histopathologic assessment that visualizes <1% of the tissue margins. A quenched protease-activated fluorescent probe 6qcNIR, which produces a signal only in the protease-rich tumor microenvironment, was topically applied to 90 specimens ex vivo immediately following excision. "Puzzle-fit" analysis was used to correlate the fluorescent images with histology. Probe-dependent fluorescent images correlated with cancer determined by conventional histology. Point-of-care fluorescent detection of skin cancer had a clinically relevant sensitivity of 0.73 and corresponding specificity of 0.88. Importantly, clinicians were effectively trained to read fluorescent images within 15 minutes with reliability and confidence, resulting in sensitivities of 62%-78% and specificities of 92%-97%. Fluorescent imaging using 6qcNIR allows 100% tumor margin assessment by generating en face images that correlate with histology and may be used to overcome the limitations of conventional bread-loaf histology. The utility of 6qcNIR was validated in a busy real-world clinical setting, and clinicians were trained to effectively read fluorescent margins with a short guided instruction, highlighting clinical adaptability. When used in conventional excision, this approach may result in higher cure rates at a lower cost by allowing same-day reexcision when needed, reducing patient anxiety and improving compliance by expediting postsurgical specimen assessment. SIGNIFICANCE: A fluorescent-probe-tumor-visualization platform was developed and validated in human keratinocyte carcinoma excision specimens that may provide simple, rapid, and global assessment of margins during skin cancer excision, allowing same-day reexcision when needed.

Evaluation of data augmentation via synthetic images for improved breast mass detection on mammograms using deep learning

We evaluated whether using synthetic mammograms for training data augmentation may reduce the effects of overfitting and increase the performance of a deep learning algorithm for breast mass detection. Synthetic mammograms were generated using in silico procedural analytic breast and breast mass modeling algorithms followed by simulated x-ray projections of the breast models into mammographic images. In silico breast phantoms containing masses were modeled across the four BI-RADS breast density categories, and the masses were modeled with different sizes, shapes, and margins. A Monte Carlo-based x-ray transport simulation code, MC-GPU, was used to project the three-dimensional phantoms into realistic synthetic mammograms. 2000 mammograms with 2522 masses were generated to augment a real data set during training. From the Curated Breast Imaging Subset of the Digital Database for Screening Mammography (CBIS-DDSM) data set, we used 1111 mammograms (1198 masses) for training, 120 mammograms (120 masses) for validation, and 361 mammograms (378 masses) for testing. We used faster R-CNN for our deep learning network with pretraining from ImageNet using the Resnet-101 architecture. We compared the detection performance when the network was trained using different percentages of the real CBIS-DDSM training set (100%, 50%, and 25%), and when these subsets of the training set were augmented with 250, 500, 1000, and 2000 synthetic mammograms. Free-response receiver operating characteristic (FROC) analysis was performed to compare performance with and without the synthetic mammograms. We generally observed an improved test FROC curve when training with the synthetic images compared to training without them, and the amount of improvement depended on the number of real and synthetic images used in training. Our study shows that enlarging the training data with synthetic samples can increase the performance of deep learning systems.

The role of digital breast tomosynthesis in breast cancer screening: a manufacturer- and metrics-specific analysis

Aim

Digital Breast Tomosynthesis (DBT), with or without Digital Mammography (DM) or Synthetic Mammography (SM), has been introduced or is under consideration for its introduction in breast cancer screening in several countries, as it has been shown that it has advantages over DM. Despite this there is no agreement on how to implement DBT in screening, and in many cases there is a lack of official guidance on the optimum usage of each commercially available system. The aim of this review is to carry out a manufacturer-specific summary of studies on the implementation of DBT in breast cancer screening.

Methods

An exhaustive literature review was undertaken to identify clinical observer studies that evaluated at least one of five common metrics: sensitivity, specificity, area under the curve (AUC) of the receiver-operating characteristics (ROC) analysis, recall rate and cancer detection rate. Four common DBT implementation methods were discussed in this review: (1) DBT, (2) DM with DBT, (3) 1-view DBT with or without 1-view DM or 2-view DM and (4) DBT with SM.

Results

A summary of 89 studies, selected from a database of 677 studies, on the assessment of the implementation of DBT in breast cancer screening is presented in tables and discussed in a manufacturer- and metric-specific approach. Much more studies were carried out using some DBT systems than others. For one implementation method of DBT by one manufacturer there is a shortage of studies, for another implementation there are conflicting results. In some cases, there is a strong agreement between studies, making the advantages and disadvantages of each system clear.

Conclusion

The optimum implementation method of DBT in breast screening, in terms of diagnostic benefit and patient radiation dose, for one manufacturer does not necessarily apply to other manufacturers.

Chest X-ray Interpretation by Radiographers Is Not Inferior to Radiologists: A Multireader, Multicase Comparison Using JAFROC (Jack-knife Alternative Free-response Receiver Operating Characteristics) Analysis

RATIONALE AND OBJECTIVES

Chest X-rays (CXR) are one of the most frequently requested imaging examinations and are fundamental to many patient pathways. The aim of this study was to investigate the diagnostic accuracy of CXR interpretation by reporting radiographers (technologists).

METHODS

A cohort of consultant radiologists (n = 10) and reporting radiographers (technologists; n = 11) interpreted a bank (n = 106) of adult CXRs that contained a range of pathologies. Jack-knife alternate free-response receiver operating characteristic (JAFROC) methodology was used to determine the performance of the observers (JAFROC v4.2). A noninferiority approach was used, with a predefined margin of clinical insignificance of 10% of average consultant radiologist diagnostic accuracy.

RESULTS

The diagnostic accuracy of the reporting radiographers (figure of merit = 0.828, 95% confidence interval 0.808-0.847) was noninferior to the consultant radiologists (figure of merit = 0.788, 95% confidence interval 0.766-0.811), P < .0001.

CONCLUSIONS

With appropriate postgraduate education, reporting radiographers are able to interpret CXRs at a level comparable to consultant radiologists.

Mammographic detection of breast cancer in a non-screening country

OBJECTIVE:

To compare the diagnostic accuracy between radiologists' from a country with and without breast cancer screening.

METHODS:

All participating radiologists gave informed consent. A test-set involving 60 mammographic cases (20 cancer and 40 non-cancer) were read by 11 radiologists from a non-screening (NS) country during a workshop in July 2016. 52 radiologists from a screening country read the same test-set at the Royal Australian and New Zealand College of Radiologists' meetings in July 2015. The screening radiologists were classified into two groups: those with less than or equal to 5 years of experience; those with more than 5 years of experience, and each group was compared to the group of NS radiologists. A Kruskal-Wallis test followed by post-hoc multiple comparisons test were used to compare measures of diagnostic accuracy among the reader groups.

RESULTS:

The diagnostic accuracy of the NS radiologists was significantly lower in terms of sensitivity [mean = 54.0; 95% confidence interval (CI) (40.0-67.0)], location sensitivity [mean = 26.0; 95% CI (16.0-37.0)], receive roperating characteristic area under curve [mean = 73.0; 95% CI (66.5-81.0)] and Jackknifefree-response receiver operating characteristics figure-of-merit [mean = 45.0; 95% CI (40.0-50.0)] when compared with the less and more experienced screening radiologists, whilst no difference in specificity [mean = 75.0; 95% CI (70.0- 81.0)] was found. No significant differences in all measured diagnostic accuracy were found between the two groups of screening radiologists.

CONCLUSION:

The mammographic performance of a group of radiologists from a country without screening program was suboptimal compared with radiologists from Australia.

ADVANCES IN KNOWLEDGE:

Identifying mammographic performance in developing countries is required to optimize breast cancer diagnosis.

One-view digital breast tomosynthesis as a stand-alone modality for breast cancer detection: do we need more?

Purpose

To compare the performance of one-view digital breast tomosynthesis (1v-DBT) to that of three other protocols combining DBT and mammography (DM) for breast cancer detection.

Materials and methods

Six radiologists, three experienced with 1v-DBT in screening, retrospectively reviewed 181 cases (76 malignant, 50 benign, 55 normal) in two sessions. First, they scored sequentially: 1v-DBT (medio-lateral oblique, MLO), 1v-DBT (MLO) + 1v-DM (cranio-caudal, CC) and two-view DM + DBT (2v-DM+2v-DBT). The second session involved only 2v-DM. Lesions were scored using BI-RADS® and level of suspiciousness (1–10). Sensitivity, specificity, receiver operating characteristic (ROC) and jack-knife alternative free-response ROC (JAFROC) were computed.

Results

On average, 1v-DBT was non-inferior to any of the other protocols in terms of JAFROC figure-of-merit, area under ROC curve, sensitivity or specificity (p>0.391). While readers inexperienced with 1v-DBT screening improved their sensitivity when adding more images (69–79 %, p=0.019), experienced readers showed similar sensitivity (76 %) and specificity (70 %) between 1v-DBT and 2v-DM+2v-DBT (p=0.482). Subanalysis by lesion type and breast density showed no difference among modalities.

Conclusion

Detection performance with 1v-DBT is not statistically inferior to 2v-DM or to 2v-DM+2v-DBT; its use as a stand-alone modality might be sufficient for readers experienced with this protocol.

Key points

• One-view breast tomosynthesis is not inferior to two-view digital mammography.

• One-view DBT is not inferior to 2-view DM plus 2-view DBT.

• Training may lead to 1v-DBT being sufficient for screening.

Detection and diagnosis of colitis on computed tomography using deep convolutional neural networks

PURPOSE

Colitis refers to inflammation of the inner lining of the colon that is frequently associated with infection and allergic reactions. In this paper, we propose deep convolutional neural networks methods for lesion-level colitis detection and a support vector machine (SVM) classifier for patient-level colitis diagnosis on routine abdominal CT scans.

METHODS

The recently developed Faster Region-based Convolutional Neural Network (Faster RCNN) is utilized for lesion-level colitis detection. For each 2D slice, rectangular region proposals are generated by region proposal networks (RPN). Then, each region proposal is jointly classified and refined by a softmax classifier and bounding-box regressor. Two convolutional neural networks, eight layers of ZF net and 16 layers of VGG net are compared for colitis detection. Finally, for each patient, the detections on all 2D slices are collected and a SVM classifier is applied to develop a patient-level diagnosis. We trained and evaluated our method with 80 colitis patients and 80 normal cases using 4 × 4-fold cross validation.

RESULTS

For lesion-level colitis detection, with ZF net, the mean of average precisions (mAP) were 48.7% and 50.9% for RCNN and Faster RCNN, respectively. The detection system achieved sensitivities of 51.4% and 54.0% at two false positives per patient for RCNN and Faster RCNN, respectively. With VGG net, Faster RCNN increased the mAP to 56.9% and increased the sensitivity to 58.4% at two false positive per patient. For patient-level colitis diagnosis, with ZF net, the average areas under the ROC curve (AUC) were 0.978 ± 0.009 and 0.984 ± 0.008 for RCNN and Faster RCNN method, respectively. The difference was not statistically significant with P = 0.18. At the optimal operating point, the RCNN method correctly identified 90.4% (72.3/80) of the colitis patients and 94.0% (75.2/80) of normal cases. The sensitivity improved to 91.6% (73.3/80) and the specificity improved to 95.0% (76.0/80) for the Faster RCNN method. With VGG net, Faster RCNN increased the AUC to 0.986 ± 0.007 and increased the diagnosis sensitivity to 93.7% (75.0/80) and specificity was unchanged at 95.0% (76.0/80).

CONCLUSION

Colitis detection and diagnosis by deep convolutional neural networks is accurate and promising for future clinical application.

Increasing Prevalence Expectation in Thoracic Radiology Leads to Overcall

RATIONALE AND OBJECTIVES

The aim of this study was to measure the effect of prevalence expectation as determined by clinical history on the diagnostic performance of radiologists during pulmonary nodule detection on adult chest radiographs.

MATERIALS AND METHODS

A multi-observer, counter-balanced study (having half the readers in each group read a different condition initially) was performed to assess the effect of abnormality expectation on experienced radiologists' performance. A total of 33 board-certified radiologists were divided into three groups and searched for evidence of malignancy on a single set of 47 postero-anterior (PA) chest radiographs, 10 of which contained a single pulmonary nodule. The radiologists were unaware of disease prevalence. Before each viewing of the same dataset, the radiologists were allocated to two of three conditions based on the differing clinical information (previous cancer, no history, visa applicant). Location sensitivity, specificity, and jack-knife free-response receiver operator characteristics figure of merit were used to compare radiologist performance between conditions.

RESULTS

A significant reduction in specificity was shown for the cancer compared to that for the visa condition (W = -41 P = 0.02). No other significant findings were demonstrated for this or the other condition comparisons. No significant difference in the performance of radiologists was noted when viewing images under the same conditions.

CONCLUSIONS

This study suggested that there is a reduction in specificity with high compared to low prevalence expectation following specific radiological contexts. A reduction in specificity can have important clinical consequences leading to unnecessary interventions. The results and their implications emphasize the caution that should be placed on providing accurate referral criteria.

Mammographic density and cancer detection: does digital imaging challenge our current understanding?

RATIONALE AND OBJECTIVES

To investigate the impact of breast density on the performance of radiologists when mammograms are digitally acquired and displayed.

MATERIALS AND METHODS

A total of 150 craniocaudal digital mammograms including 75 cases with cancer were examined by 14 radiologists divided into two groups: those who read more (six) and less (eight) than 2000 mammograms per year. Cases were classified as low or high mammographic density. For both types of cases, detection of cancers within and outside the dense fibroglandular tissue was investigated. The performance of radiologist was measured using jack-knife free-response receiver operating characteristic (JAFROC) figure of merit (FOM).

RESULTS

Radiologists with over 2000 annual reads had significantly higher JAFROC FOM (P = .03) for high (0.76) mammographic density compared to low (0.70) mammographic density cases. When lesions overlaid the fibroglandular tissue, cases with high mammographic density compared to low mammographic density displayed increased location sensitivity for all radiologists (P = .03) and for those radiologists reading more than 2000 mammograms annually (P = .04), whereas JAFROC FOMs increased for all radiologists (P = .05). No significant changes were observed when the lesion was outside the fibroglandular region.

CONCLUSIONS

Increased mammographic density improves the performance of experienced radiologists when using digital mammograms. This finding, which does not align with those previously reported for film screen systems, may be because of windowing/leveling opportunities available with digital images.

Discovering intrinsic properties of human observers' visual search and mathematical observers' scanning

There is a lack of consensus in measuring observer performance in search tasks. To pursue a consensus, we set our goal to obtain metrics that are practical, meaningful, and predictive. We consider a metric practical if it can be implemented to measure human and computer observers' performance. To be meaningful, we propose to discover intrinsic properties of search observers and formulate the metrics to characterize these properties. If the discovered properties allow verifiable predictions, we consider them predictive. We propose a theory and a conjecture toward two intrinsic properties of search observers: rationality in classification as measured by the location-known-exactly (LKE) receiver operating characteristic (ROC) curve and location uncertainty as measured by the effective set size (M^*). These two properties are used to develop search models in both single-response and free-response search tasks. To confirm whether these properties are "intrinsic," we investigate their ability in predicting search performance of both human and scanning channelized Hotelling observers. In particular, for each observer, we designed experiments to measure the LKE-ROC curve and M^*, which were then used to predict the same observer's performance in other search tasks. The predictions were then compared to the experimentally measured observer performance. Our results indicate that modeling the search performance using the LKE-ROC curve and M^* leads to successful predictions in most cases.

Incremental value of liver MR imaging in patients with potentially curable colorectal hepatic metastasis detected at CT: a prospective comparison of diffusion-weighted imaging, gadoxetic acid-enhanced MR imaging, and a combination of both MR techniques

PURPOSE

To prospectively compare diagnostic performance of diffusion-weighted (DW) imaging, gadoxetic acid-enhanced magnetic resonance (MR) imaging, both techniques combined (combined MR imaging), and computed tomography (CT) for detecting colorectal hepatic metastases and evaluate incremental value of MR for patients with potentially curable colorectal hepatic metastases detected with CT.

MATERIALS AND METHODS

In this institutional review board-approved prospective study, with informed consent, 51 patients (39 men, 12 women; mean age, 62 years) with potentially resectable hepatic metastases detected with CT underwent liver MR, including DW imaging and gadoxetic acid-enhanced MR. Two independent readers reviewed DW, gadoxetic acid-enhanced, combined MR, and CT image sets to detect hepatic metastases. The figure-of-merit (FOM) value representing overall diagnostic performance, sensitivity, and positive predictive value (PPV) for each image set were analyzed by using free-response receiver operating characteristic analysis and generalized estimating equations.

RESULTS

There were 104 hepatic metastases in 47 patients. The pooled FOM values, sensitivities, and PPVs of combined MR (FOM value, 0.93; sensitivity, 98%; and PPV, 88%) and gadoxetic acid-enhanced MR (FOM value, 0.92; sensitivity, 95%; and PPV, 90%) were significantly higher than those of CT (FOM value, 0.82; sensitivity, 85%; and PPV, 73%) (P < .006). The pooled FOM value and sensitivity of combined MR (FOM value, 0.92; sensitivity, 95%) was also significantly higher than that of DW imaging (FOM value, 0.82; sensitivity, 79%) for metastases (≤1-cm diameter) (P ≤ .003). DW imaging showed significantly higher pooled sensitivity (79%) and PPV (60%) than CT (sensitivity, 50%; PPV, 33%) for the metastases (≤1-cm diameter) (P ≤ .004). In 47 patients with hepatic metastases, combined MR depicted more metastases than CT in 10 and 14 patients, respectively, according to both readers.

CONCLUSION

Gadoxetic acid-enhanced MR and combined MR are more accurate than CT in detecting colorectal hepatic metastases, have an incremental value when added to CT alone for detecting additional metastases, and can be routinely performed in patients with potentially curable hepatic metastases detected with CT.

Model observers in medical imaging research

Model observers play an important role in the optimization and assessment of imaging devices. In this review paper, we first discuss the basic concepts of model observers, which include the mathematical foundations and psychophysical considerations in designing both optimal observers for optimizing imaging systems and anthropomorphic observers for modeling human observers. Second, we survey a few state-of-the-art computational techniques for estimating model observers and the principles of implementing these techniques. Finally, we review a few applications of model observers in medical imaging research.

Estimating the relative utility of screening mammography

BACKGROUND

The concept of diagnostic utility is a fundamental component of signal detection theory, going back to some of its earliest works. Attaching utility values to the various possible outcomes of a diagnostic test should, in principle, lead to meaningful approaches to evaluating and comparing such systems. However, in many areas of medical imaging, utility is not used because it is presumed to be unknown.

METHODS

In this work, we estimate relative utility (the utility benefit of a detection relative to that of a correct rejection) for screening mammography using its known relation to the slope of a receiver operating characteristic (ROC) curve at the optimal operating point. The approach assumes that the clinical operating point is optimal for the goal of maximizing expected utility and therefore the slope at this point implies a value of relative utility for the diagnostic task, for known disease prevalence. We examine utility estimation in the context of screening mammography using the Digital Mammographic Imaging Screening Trials (DMIST) data.

RESULTS

We show how various conditions can influence the estimated relative utility, including characteristics of the rating scale, verification time, probability model, and scope of the ROC curve fit. Relative utility estimates range from 66 to 227.

CONCLUSIONS

We argue for one particular set of conditions that results in a relative utility estimate of 162 (±14%). This is broadly consistent with values in screening mammography determined previously by other means. At the disease prevalence found in the DMIST study (0.59% at 365-day verification), optimal ROC slopes are near unity, suggesting that utility-based assessments of screening mammography will be similar to those found using Youden's index.

Breast tomosynthesis and digital mammography: a comparison of diagnostic accuracy

OBJECTIVE

Our aim was to compare the ability of radiologists to detect breast cancers using one-view breast tomosynthesis (BT) and two-view digital mammography (DM) in an enriched population of diseased patients and benign and/or healthy patients.

METHODS

All participants gave informed consent. The BT and DM examinations were performed with about the same average glandular dose to the breast. The study population comprised patients with subtle signs of malignancy seen on DM and/or ultrasonography. Ground truth was established by pathology, needle biopsy and/or by 1-year follow-up by mammography, which retrospectively resulted in 89 diseased breasts (1 breast per patient) with 95 malignant lesions and 96 healthy or benign breasts. Two experienced radiologists, who were not participants in the study, determined the locations of the malignant lesions. Five radiologists, experienced in mammography, interpreted the cases independently in a free-response study. The data were analysed by the receiver operating characteristic (ROC) and jackknife alternative free-response ROC (JAFROC) methods, regarding both readers and cases as random effects.

RESULTS

The diagnostic accuracy of BT was significantly better than that of DM (JAFROC: p=0.0031, ROC: p=0.0415). The average sensitivity of BT was higher than that of DM (∼90% vs ∼79%; 95% confidence interval of difference: 0.036, 0.108) while the average false-positive fraction was not significantly different (95% confidence interval of difference: -0.117, 0.010).

CONCLUSION

The diagnostic accuracy of BT was superior to DM in an enriched population.

A perceptually relevant channelized joint observer (PCJO) for the detection-localization of parametric signals

Many numerical observers have been proposed in the framework of task-based approach for medical image quality assessment. However, the existing numerical observers are still limited in diagnostic tasks: the detection task has been largely studied, while the localization task concerning one signal has been little studied and the localization of multiple signals has not been studied yet. In addition, most existing numerical observers need a priori knowledge about all the parameters of the underdetection signals, while only a few of them need at least two signal parameters. In this paper, we propose a novel numerical observer called the perceptually relevant channelized joint observer (PCJO), which cannot only detect but also localize multiple signals with unknown amplitude, orientation, size and location. We validated the PCJO for predicting human observer task performance by conducting a clinically relevant free-response subjective experiment in which six radiologists (including two experts) had to detect and localize multiple sclerosis (MS) lesions on magnetic resonance (MR) images. By using the jackknife alternative free-response operating characteristic (JAFROC) as the figure of merit (FOM), the detection-localization task performance of the PCJO was evaluated and then compared to that of the radiologists and two other numerical observers--channelized hotelling observer (CHO) and Goossenss CHO for detecting asymmetrical signals with random orientations. Overall, the results show that the PCJO performance was closer to that of the experts than to that of the other radiologists. The JAFROC1 FOMs of the PCJO (around 0.75) are not significantly different from those of the two experts (0.7672 and 0.7110), while the JAFROC1 FOMs of the numerical observers mentioned above (always over 0.84) outperform those of the experts. This indicates that the PCJO is a promising method for predicting radiologists' performance in the joint detection-localization task.

Learning aspects and potential pitfalls regarding detection of pulmonary nodules in chest tomosynthesis and proposed related quality criteria

BACKGROUND

In chest tomosynthesis, low-dose projections collected over a limited angular range are used for reconstruction of an arbitrary number of section images of the chest, resulting in a moderately increased radiation dose compared to chest radiography.

PURPOSE

To investigate the effects of learning with feedback on the detection of pulmonary nodules for observers with varying experience of chest tomosynthesis, to identify pitfalls regarding detection of pulmonary nodules, and present suggestions for how to avoid them, and to adapt the European quality criteria for chest radiography and computed tomography (CT) to chest tomosynthesis.

MATERIAL AND METHODS

Six observers analyzed tomosynthesis cases for presence of nodules in a jackknife alternative free-response receiver-operating characteristics (JAFROC) study. CT was used as reference. The same tomosynthesis cases were analyzed before and after learning with feedback, which included a collective learning session. The difference in performance between the two readings was calculated using the JAFROC figure of merit as principal measure of detectability.

RESULTS

Significant improvement in performance after learning with feedback was found only for observers inexperienced in tomosynthesis. At the collective learning session, localization of pleural and subpleural nodules or structures was identified as the main difficulty in analyzing tomosynthesis images.

CONCLUSION

The results indicate that inexperienced observers can reach a high level of performance regarding nodule detection in tomosynthesis after learning with feedback and that the main problem with chest tomosynthesis is related to the limited depth resolution.

Detection of clustered microcalcifications using spatial point process modeling

In this work we propose a spatial point process (SPP) approach to improve the detection accuracy of clustered microcalcifications (MCs) in mammogram images. The conventional approach to MC detection has been to first detect the individual MCs in an image independently, which are subsequently grouped into clusters. Our proposed approach aims to exploit the spatial distribution among the different MCs in a mammogram image (i.e. MCs tend to appear in small clusters) directly during the detection process. We model the MCs by a marked point process (MPP) in which spatially neighboring MCs interact with each other. The MCs are then simultaneously detected through maximum a posteriori (MAP) estimation of the model parameters associated with the MPP process. The proposed approach was evaluated with a dataset of 141 clinical mammograms from 66 cases, and the results show that it could yield improved detection performance compared to a recently proposed support vector machine (SVM) detector. In particular, the proposed approach achieved a sensitivity of about 90% with the FP rate at around 0.5 clusters per image, compared to about 83% for the SVM; the performance of the proposed approach was also demonstrated to be more stable over different compositions of the test images.

X-ray tomosynthesis: a review of its use for breast and chest imaging

Tomosynthesis is a three-dimensional imaging technique based on the reconstruction of several planar radiographs. During the image acquisition in tomosynthesis, the X-ray tube moves around the detector which is often stationary, and a number of projection images are taken from different angles. Individual slices from the reconstructed volume can be studied. With the effective reduction of the visibility of the overlapping normal tissue, the detection of pathological lesions is improved when compared with projection radiography. Up to now, tomosynthesis has mainly been used for breast and chest examinations and, to some extent, also for orthopaedic, angiographic and dental investigations. For chest, tomosynthesis is used as an alternative to computed tomography with significantly lower cost and radiation dose to the patient. Breast tomosynthesis has, in several studies, proved to be an effective tool for improving detection of breast lesions. As tomosynthesis has many properties that make it suitable as a modality for screening, including good diagnostic performance, short examination time and low radiation dose, it is a strong competitor to the current gold standard breast screening modality, i.e. mammography. In this paper, the principles of tomosynthesis will be presented as well as a few clinical studies showing the potential role of tomosynthesis in clinical routine examinations.

VIEWDEX: an efficient and easy-to-use software for observer performance studies

The development of investigation techniques, image processing, workstation monitors, analysing tools etc. within the field of radiology is vast, and the need for efficient tools in the evaluation and optimisation process of image and investigation quality is important. ViewDEX (Viewer for Digital Evaluation of X-ray images) is an image viewer and task manager suitable for research and optimisation tasks in medical imaging. ViewDEX is DICOM compatible and the features of the interface (tasks, image handling and functionality) are general and flexible. The configuration of a study and output (for example, answers given) can be edited in any text editor. ViewDEX is developed in Java and can run from any disc area connected to a computer. It is free to use for non-commercial purposes and can be downloaded from http://www.vgregion.se/sas/viewdex. In the present work, an evaluation of the efficiency of ViewDEX for receiver operating characteristic (ROC) studies, free-response ROC (FROC) studies and visual grading (VG) studies was conducted. For VG studies, the total scoring rate was dependent on the number of criteria per case. A scoring rate of approximately 150 cases h(-1) can be expected for a typical VG study using single images and five anatomical criteria. For ROC and FROC studies using clinical images, the scoring rate was approximately 100 cases h(-1) using single images and approximately 25 cases h(-1) using image stacks ( approximately 50 images case(-1)). In conclusion, ViewDEX is an efficient and easy-to-use software for observer performance studies.

Dose reduction and its influence on diagnostic accuracy and radiation risk in digital mammography: an observer performance study using an anthropomorphic breast phantom

This study aimed to investigate the effect of dose reduction on diagnostic accuracy and radiation risk in digital mammography. Simulated masses and microcalcifications were positioned in an anthropomorphic breast phantom. Thirty digital images, 14 with lesions, 16 without, were acquired of the phantom using a Mammomat Novation (Siemens, Erlangen, Germany) at each of three dose levels. These corresponded to 100%, 50% and 30% of the normally used average glandular dose (AGD; 1.3 mGy for a standard breast). Eight observers interpreted the 90 unprocessed images in a free response study, and the data were analysed with the jackknife free response receiver operating characteristic (JAFROC) method. Observer performance was assessed using the JAFROC figure of merit (FOM). The benefit of radiation risk reduction was estimated based on several risk models. There was no statistically significant difference in performance, as described by the FOM, between the 100% and the 50% dose levels. However, the FOMs for both the 100% and the 50% dose were significantly different from the corresponding quantity for the 30% dose level (F-statistic = 4.95, p-value = 0.01). A dose reduction of 50% would result in three to nine fewer breast cancer fatalities per 100,000 women undergoing annual screening from the age of 40 to 49 years. The results of the study indicate a possibility of reducing the dose to the breast to half the dose level currently used. This has to be confirmed in clinical studies, and possible differences depending on lesion type should be examined further.

Dose dependence of mass and microcalcification detection in digital mammography: free response human observer studies

The purpose of this study was to evaluate the effect of dose reduction in digital mammography on the detection of two lesion types-malignant masses and clusters of microcalcifications. Two free-response observer studies were performed-one for each lesion type. Ninety screening images were retrospectively selected; each image was originally acquired under automatic exposure conditions, corresponding to an average glandular dose of 1.3 mGy for a standard breast (50 mm compressed breast thickness with 50% glandularity). For each study, one to three simulated lesions were added to each of 40 images (abnormals) while 50 were kept without lesions (normals). Two levels of simulated system noise were added to the images yielding two new image sets, corresponding to simulated dose levels of 50% and 30% of the original images (100%). The manufacturer's standard display processing was subsequently applied to all images. Four radiologists experienced in mammography evaluated the images by searching for lesions and marking and assigning confidence levels to suspicious regions. The search data were analyzed using jackknife free-response (JA-FROC) methodology. For the detection of masses, the mean figure-of-merit (FOM) averaged over all readers was 0.74, 0.71, and 0.68 corresponding to dose levels of 100%, 50%, and 30%, respectively. These values were not statistically different from each other (F= 1.67, p=0.19) but showed a decreasing trend. In contrast, in the microcalcification study the mean FOM was 0.93, 0.67, and 0.38 for the same dose levels and these values were all significantly different from each other (F = 109.84, p < 0.0001). The results indicate that lowering the present dose level by a factor of two compromised the detection of microcalcifications but had a weaker effect on mass detection.