Accuracy and Interobserver Variability in Reporting of PI-RADS Version 2

More than Meets the Eye: Using Textural Analysis and Artificial Intelligence as Decision Support Tools in Prostate Cancer Diagnosis—A Systematic Review

(1) Introduction: Multiparametric magnetic resonance imaging (mpMRI) is the main imagistic tool employed to assess patients suspected of harboring prostate cancer (PCa), setting the indication for targeted prostate biopsy. However, both mpMRI and targeted prostate biopsy are operator dependent. The past decade has been marked by the emerging domain of radiomics and artificial intelligence (AI), with extended application in medical diagnosis and treatment processes. (2) Aim: To present the current state of the art regarding decision support tools based on texture analysis and AI for the prediction of aggressiveness and biopsy assistance. (3) Materials and Methods: We performed literature research using PubMed MeSH, Scopus and WoS (Web of Science) databases and screened the retrieved papers using PRISMA principles. Articles that addressed PCa diagnosis and staging assisted by texture analysis and AI algorithms were included. (4) Results: 359 papers were retrieved using the keywords “prostate cancer”, “MRI”, “radiomics”, “textural analysis”, “artificial intelligence”, “computer assisted diagnosis”, out of which 35 were included in the final review. In total, 24 articles were presenting PCa diagnosis and prediction of aggressiveness, 7 addressed extracapsular extension assessment and 4 tackled computer-assisted targeted prostate biopsies. (5) Conclusions: The fusion of radiomics and AI has the potential of becoming an everyday tool in the process of diagnosis and staging of the prostate malignancies.

Interobserver agreement of Prostate Imaging–Reporting and Data System (PI-RADS–v2)

Background

A retrospective study was conducted on 71 consecutive patients with suspected prostate cancer (PCa) with a mean age of 56 years and underwent mp-MRI of the prostate at 3 Tesla MRI. Two readers recognized all prostatic lesions, and each lesion had a score according to Prostate Imaging–Reporting and Data System version 2 (PI-RADS-v2).

Purpose of the study

To evaluate the interobserver agreement of PI-RADS-v2 in characterization of prostatic lesions using multiparametric MRI (mp-MRI) at 3 Tesla MRI.

Results

The overall interobserver agreement of PI-RADS-v2 for both zones was excellent (k = 0.81, percent agreement = 94.9%). In the peripheral zone (PZ) lesions are the interobserver agreement for PI-RADS II (k = 0.78, percent agreement = 83.9%), PI-RADS III (k = 0.66, percent agreement = 91.3 %), PI-RADS IV (k = 0.69, percent agreement = 93.5%), and PI-RADS V (k = 0.91, percent agreement = 95.7 %). In the transitional zone (TZ) lesions are the interobserver agreement for PI-RADS I (k = 0.98, percent of agreement = 96%), PI-RADS II (k = 0.65, percent agreement = 96%), PI-RADS III (k = 0.65, percent agreement = 88%), PI-RADS IV (k = 0.83, percent agreement = 96%), and PI-RADS V (k = 0.82, percent agreement = 92%).

Conclusion

We concluded that PI-RADS-v2 is a reliable and a reproducible imaging modality for the characterization of prostatic lesions and detection of PCa.

Implementation and design of artificial intelligence in abdominal imaging

Artificial intelligence is a technique that holds promise for helping radiologists improve the care of our patients. At the same time, implementation decisions we make now can have a long-lasting effect on patient outcomes. In the following article, we discuss four areas with unique considerations for implementation of AI: bias, trust, risk, and design. In each section, we highlight applications of AI to abdominal imaging and prostate cancer specifically.

PI-RADS Versions 2 and 2.1: Interobserver Agreement and Diagnostic Performance in Peripheral and Transition Zone Lesions Among Six Radiologists

BACKGROUND. PI-RADS version 2.1 (v2.1) modifications primarily address transition zone (TZ) interpretation. The revisions also impact peripheral zone (PZ) interpretation, which has received less attention. OBJECTIVE. The purpose of this study was to compare interobserver agreement of PI-RADS version 2 (v2) and v2.1 in the prostate PZ and TZ and perform a pilot comparison of their diagnostic performance in the two zones. METHODS. Six radiologists with varying experience retrospectively assessed 80 prostate lesions (40 PZ, 40 TZ) on MRI in separate sessions for PI-RADS v2 and v2.1. Interobserver agreement was assessed using Conger kappa (κ). For 50 lesions with pathology data, average AUC for detecting clinically significant cancer was compared between versions using multireader multicase statistical methods. Error variance and covariance results informed post hoc power analysis. RESULTS. Interobserver agreement for PI-RADS category 4 or greater was higher for version 2.1 (κ = 0.64) than version 2 (κ = 0.51) in the PZ, but similar for version 2 (κ = 0.64) and version 2.1 (κ = 0.60) in the TZ. The PI-RADS v2.1 DWI descriptor "linear/wedge-shaped" had higher agreement than its predecessor version 2 descriptor "indistinct hypointense" (κ = 0.52 vs κ = 0.18) and yielded 14 more true-negative versus five more false-negative interpretations. The ADC signal descriptor "markedly hypointense," for which only version 2.1 provides a specific definition, had lower agreement in version 2.1 (κ = 0.26) than version 2 (κ = 0.52). Modified TZ T2-weighted category 2 descriptors in version 2.1 had fair agreement (κ = 0.21), and agreement for PI-RADS category 2 in the TZ was lower in version 2.1 (κ = 0.31) than version 2 (κ = 0.57). DWI upgraded a TZ lesion category from 2 to 3 in four patients, detecting two additional cancers. Average AUC was not different between versions 2 and 2.1 for the PZ (AUC, 0.81 vs 0.85; p = .24) or the TZ (AUC, 0.69 vs 0.69; p = .94), though among experienced readers AUC was higher for version 2.1 than version 2 for the PZ (0.91 vs 0.82; p = .001). Overall performance comparison had sufficient power (0.8) to detect a 0.085 difference in AUC. CONCLUSION. Interobserver agreement improved using PI-RADS v2.1 in the PZ but not the TZ. Diagnostic performance improved using version 2.1 only in the PZ for experienced readers. Specific version 2.1 modifications yielded mixed results. CLINICAL IMPACT. The impact of PI-RADS v2.1 in the PZ is notable given the emphasis on version 2.1 TZ modifications. The findings suggest areas in which additional modification could further improve interobserver agreement and performance.

Validity and Reproducibility of the ADNEX MR Scoring System in the Diagnosis of Sonographically Indeterminate Adnexal Masses

BACKGROUND

The diagnosis of sonographically indeterminate adnexal masses (AM) signifies a major challenge in clinical practice. Early detection and characterization have increased the need for accurate imaging evaluation before treatment.

PURPOSE

To assess the validity and reproducibility of the ADNEX MR Scoring system in the diagnosis of sonographically indeterminate AM.

STUDY TYPE

A prospective multicenter study.

POPULATION

In all, 531 women (mean age, 44 ± 11.2 years; range, 21-79 years) with 572 sonographically indeterminate AM.

FIELD STRENGTH/SEQUENCE

1.5T/precontrast T₁ -weighted imaging (WI) fast spin echo (FSE) (in-phase and out-of-phase, with and without fat suppression); T₂ -WI FSE; diffusion-WI single-shot echo planner with b-values of 0 and 1000 s/mm² ; and dynamic contrast-enhanced perfusion T₁ -WI liver acquisition with volume acceleration (LAVA).

ASSESSMENT

All MRI examinations were evaluated by three radiologists, and the AM were categorized into five scores based on the ADNEX MR Scoring system. Score 1: no AM; 2: benign AM; 3: probably benign AM; 4: indeterminate AM; 5: probably malignant AM. Histopathology and imaging follow-up were used as the standard references for evaluating the validity of the ADNEX MR Scoring system for detecting ovarian malignancy.

STATISTICAL TESTS

Four-fold table test, kappa statistics (κ), and receiver operating characteristic (ROC) curve.

RESULTS

In all, 136 (23.8%) AM were malignant, and 436 (76.2%) were benign. Of the 350 AM classified as score 2, one (0.3%) was malignant; of the 62 AM classified as score 3, six (9.7%) were malignant; of the 73 AM classified as score 4, 43 (58.9%) were malignant; and of the 87 AM categorized as score 5, 86 (98.9%) were malignant. The best cutoff value for predicting malignant AM was score >3 with sensitivity and specificity of 92.9% and 94.9%, respectively. The interreader agreement of the ADNEX MR Scoring was very good (κ = 0.861).

DATA CONCLUSION

The current study supports the high validity and reproducibility of the ADNEX MR Scoring system for the diagnosis of sonographically indeterminate AM.

LEVEL OF EVIDENCE

1 TECHNICAL EFFICACY STAGE: 2.

Interreader Variability of Prostate Imaging Reporting and Data System Version 2 in Detecting and Assessing Prostate Cancer Lesions at Prostate MRI

OBJECTIVE. The purpose of this study was to evaluate agreement among radiologists in detecting and assessing prostate cancer at multiparametric MRI using Prostate Imaging Reporting and Data System version 2 (PI-RADSv2). MATERIALS AND METHODS. Treatment-naïve patients underwent 3-T multipara-metric MRI between April 2012 and June 2015. Among the 163 patients evaluated, 110 underwent prostatectomy after MRI and 53 had normal MRI findings and transrectal ultrasound-guided biopsy results. Nine radiologists participated (three each with high, intermediate, and low levels of experience). Readers interpreted images of 58 patients on average (range, 56-60) using PI-RADSv2. Prostatectomy specimens registered to MRI were ground truth. Interob-server agreement was evaluated with the index of specific agreement for lesion detection and kappa and proportion of agreement for PI-RADS category assignment. RESULTS. The radiologists detected 336 lesions. Sensitivity for index lesions was 80.9% (95% CI, 75.1-85.9%), comparable across reader experience (p = 0.392). Patient-level specificity was experience dependent; highly experienced readers had 84.0% specificity versus 55.2% for all others (p < 0.001). Interobserver agreement was excellent for detecting index lesions (index of specific agreement, 0.871; 95% CI, 0.798-0.923). Agreement on PI-RADSv2 category assignment of index lesions was moderate (κ = 0.419; 95% CI, 0.238-0.595). For individual category assignments, proportion of agreement was slight for PI-RADS category 3 (0.208; 95% CI, 0.086-0.284) but substantial for PI-RADS category 4 (0.674; 95% CI, 0.540-0.776). However, proportion of agreement for T2-weighted PI-RADS 4 in the transition zone was 0.250 (95% CI, 0.108-0.372). Proportion of agreement for category assignment of index lesions on dynamic contrast-enhanced MR images was 0.822 (95% CI, 0.728-0.903), on T2-weighted MR images was 0.515 (95% CI, 0.430-0623), and on DW images was 0.586 (95% CI, 0.495-0.682). Proportion of agreement for dominant lesion was excellent (0.828; 95% CI, 0.742-0.913). CONCLUSION. Radiologists across experience levels had excellent agreement for detecting index lesions and moderate agreement for category assignment of lesions using PI-RADS. Future iterations of PI-RADS should clarify PI-RADS 3 and PI-RADS 4 in the transition zone.

All over the map: An interobserver agreement study of tumor location based on the PI-RADSv2 sector map

BACKGROUND

Prostate imaging reporting and data system version 2 (PI-RADSv2) recommends a sector map for reporting findings of prostate cancer mulitparametric MRI (mpMRI). Anecdotally, radiologists may demonstrate inconsistent reproducibility with this map.

PURPOSE

To evaluate interobserver agreement in defining prostate tumor location on mpMRI using the PI-RADSv2 sector map.

STUDY TYPE

Retrospective.

POPULATION

Thirty consecutive patients who underwent mpMRI between October, 2013 and March, 2015 and who subsequently underwent prostatectomy with whole-mount processing.

FIELD STRENGTH

3T mpMRI with T2 W, diffusion-weighted imaging (DWI) (apparent diffusion coefficient [ADC] and b-2000), dynamic contrast-enhanced (DCE).

ASSESSMENT

Six radiologists (two high, two intermediate, and two low experience) from six institutions participated. Readers were blinded to lesion location and detected up to four lesions as per PI-RADSv2 guidelines. Readers marked the long-axis of lesions, saved screen-shots of each lesion, and then marked the lesion location on the PI-RADSv2 sector map. Whole-mount prostatectomy specimens registered to the MRI served as ground truth. Index lesions were defined as the highest grade lesion or largest lesion if grades were equivalent.

STATISTICAL TEST

Agreement was calculated for the exact, overlap, and proportion of agreement.

RESULTS

Readers detected an average of 1.9 lesions per patient (range 1.6-2.3). 96.3% (335/348) of all lesions for all readers were scored PI-RADS ≥3. Readers defined a median of 2 (range 1-18) sectors per lesion. Agreement for detecting index lesions by screen shots was 83.7% (76.1%-89.9%) vs. 71.0% (63.1-78.3%) overlap agreement on the PI-RADS sector map (P < 0.001). Exact agreement for defining sectors of detected index lesions was only 21.2% (95% confidence interval [CI]: 14.4-27.7%) and rose to 49.0% (42.4-55.3%) when overlap was considered. Agreement on defining the same level of disease (ie, apex, mid, base) was 61.4% (95% CI 50.2-71.8%).

DATA CONCLUSION

Readers are highly likely to detect the same index lesion on mpMRI, but exhibit poor reproducibility when attempting to define tumor location on the PI-RADSv2 sector map. The poor agreement of the PI-RADSv2 sector map raises concerns its utility in clinical practice.

LEVEL OF EVIDENCE

3 Technical Efficacy: Stage 2 J. MAGN. RESON. IMAGING 2018;48:482-490.

Coronary artery disease reporting and data system (CAD-RADSTM): Inter-observer agreement for assessment categories and modifiers

BACKGROUND

The Coronary Artery Disease Reporting and Data System (CAD-RADS) provides a lexicon and standardized reporting system for coronary CT angiography.

OBJECTIVES

To evaluate inter-observer agreement of the CAD-RADS among an panel of early career and expert readers.

METHODS

Four early career and four expert cardiac imaging readers prospectively and independently evaluated 50 coronary CT angiography cases using the CAD-RADS lexicon. All readers assessed image quality using a five-point Likert scale, with mean Likert score ≥4 designating high image quality, and <4 designating moderate/low image quality. All readers were blinded to medical history and invasive coronary angiography findings. Inter-observer agreement for CAD-RADS assessment categories and modifiers were assessed using intra-class correlation (ICC) and Fleiss' Kappa (κ).The impact of reader experience and image quality on inter-observer agreement was also examined.

RESULTS

Inter-observer agreement for CAD-RADS assessment categories was excellent (ICC 0.958, 95% CI 0.938-0.974, p < 0.0001). Agreement among expert readers (ICC 0.925, 95% CI 0.884-0.954) was marginally stronger than for early career readers (ICC 0.904, 95% CI 0.852-0.941), both p < 0.0001. High image quality was associated with stronger agreement than moderate image quality (ICC 0.944, 95% CI 0.886-0.974 vs. ICC 0.887, 95% CI 0.775-0.95, both p < 0.0001). While excellent inter-observer agreement was observed for modifiers S (stent) and G (bypass graft) (both κ = 1.0), only fair agreement (κ = 0.40) was observed for modifier V (high risk plaque).

CONCLUSION

Inter-observer reproducibility of CAD-RADS assessment categories and modifiers is excellent, except for high-risk plaque (modifier V) which demonstrates fair agreement. These results suggest CAD-RADS is feasible for clinical implementation.