An evolutionary algorithm for the direct optimization of covariate balance between nonrandomized populations

Matching reduces confounding bias in comparing the outcomes of nonrandomized patient populations by removing systematic differences between them. Under very basic assumptions, propensity score (PS) matching can be shown to eliminate bias entirely in estimating the average treatment effect on the treated. In practice, misspecification of the PS model leads to deviations from theory and matching quality is ultimately judged by the observed post-matching balance in baseline covariates. Since covariate balance is the ultimate arbiter of successful matching, we argue for an approach to matching in which the success criterion is explicitly specified and describe an evolutionary algorithm to directly optimize an arbitrary metric of covariate balance. We demonstrate the performance of the proposed method using a simulated dataset of 275,000 patients and 10 matching covariates. We further apply the method to match 250 patients from a recently completed clinical trial to a pool of more than 160,000 patients identified from electronic health records on 101 covariates. In all cases, we find that the proposed method outperforms PS matching as measured by the specified balance criterion. We additionally find that the evolutionary approach can perform comparably to another popular direct optimization technique based on linear integer programming, while having the additional advantage of supporting arbitrary balance metrics. We demonstrate how the chosen balance metric impacts the statistical properties of the resulting matched populations, emphasizing the potential impact of using nonlinear balance functions in constructing an external control arm. We release our implementation of the considered algorithms in Python.

Identification of International Society on Thrombosis and Haemostasis major and clinically relevant non-major bleed events from electronic health records: a novel algorithm to enhance data utilisation from real-world sources

Introduction

In randomised controlled trials (RCTs), bleeding outcomes are often assessed using definitions provided by the International Society on Thrombosis and Haemostasis (ISTH). Information relating to bleeding events in real-world evidence (RWE) sources are not identified using these definitions. To assist with accurate comparisons between clinical trials and real-world studies, algorithms are required for the identification of ISTH-defined bleeding events in RWE sources.

Objectives

To present a novel algorithm to identify ISTH-defined major and clinically-relevant non-major (CRNM) bleeding events in a US Electronic Health Record (EHR) database.

Methods

The ISTH definition for major bleeding was divided into three subclauses: fatal bleeds, critical organ bleeds and symptomatic bleeds associated with haemoglobin reductions. Data elements from EHRs required to identify patients fulfilling these subclauses (algorithm components) were defined according to International Classification of Diseases, 9th and 10th Revisions, Clinical Modification disease codes that describe key bleeding events. Other data providing context to bleeding severity included in the algorithm were: 'interaction type' (diagnosis in the inpatient or outpatient setting), 'position' (primary/discharge or secondary diagnosis), haemoglobin values from laboratory tests, blood transfusion codes and mortality data.

Results

In the final algorithm, the components were combined to align with the subclauses of ISTH definitions for major and CRNM bleeds. A matrix was proposed to guide identification of ISTH bleeding events in the EHR database. The matrix categorises bleeding events by combining data from algorithm components, including: diagnosis codes, 'interaction type', 'position', decreases in haemoglobin concentrations (≥ 2 g/dL over 48 hours) and mortality.

Conclusions

The novel algorithm proposed here identifies ISTH major and CRNM bleeding events that are commonly investigated in RCTs in a real-world EHR data source. This algorithm could facilitate comparison between the frequency of bleeding outcomes recorded in clinical trials and RWE. Validation of algorithm performance is in progress.

Interactive Explainable Deep Learning Model Informs Prostate Cancer Diagnosis at MRI

Background Clinically significant prostate cancer (PCa) diagnosis at MRI requires accurate and efficient radiologic interpretation. Although artificial intelligence may assist in this task, lack of transparency has limited clinical translation. Purpose To develop an explainable artificial intelligence (XAI) model for clinically significant PCa diagnosis at biparametric MRI using Prostate Imaging Reporting and Data System (PI-RADS) features for classification justification. Materials and Methods This retrospective study included consecutive patients with histopathologic analysis-proven prostatic lesions who underwent biparametric MRI and biopsy between January 2012 and December 2017. After image annotation by two radiologists, a deep learning model was trained to detect the index lesion; classify PCa, clinically significant PCa (Gleason score ≥ 7), and benign lesions (eg, prostatitis); and justify classifications using PI-RADS features. Lesion- and patient-based performance were assessed using fivefold cross validation and areas under the receiver operating characteristic curve. Clinical feasibility was tested in a multireader study and by using the external PROSTATEx data set. Statistical evaluation of the multireader study included Mann-Whitney U and exact Fisher-Yates test. Results Overall, 1224 men (median age, 67 years; IQR, 62-73 years) had 3260 prostatic lesions (372 lesions with Gleason score of 6; 743 lesions with Gleason score of ≥ 7; 2145 benign lesions). XAI reliably detected clinically significant PCa in internal (area under the receiver operating characteristic curve, 0.89) and external test sets (area under the receiver operating characteristic curve, 0.87) with a sensitivity of 93% (95% CI: 87, 98) and an average of one false-positive finding per patient. Accuracy of the visual and textual explanations of XAI classifications was 80% (1080 of 1352), confirmed by experts. XAI-assisted readings improved the confidence (4.1 vs 3.4 on a five-point Likert scale; P = .007) of nonexperts in assessing PI-RADS 3 lesions, reducing reading time by 58 seconds (P = .009). Conclusion The explainable AI model reliably detected and classified clinically significant prostate cancer and improved the confidence and reading time of nonexperts while providing visual and textual explanations using well-established imaging features. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Chapiro in this issue.

Inferring FDG-PET-positivity of lymph node metastases in proven lung cancer from contrast-enhanced CT using radiomics and machine learning

Background

We evaluated the role of radiomics applied to contrast-enhanced computed tomography (CT) in the detection of lymph node (LN) metastases in patients with known lung cancer compared to ¹⁸F-fluorodeoxyglucose positron emission tomography (PET)/CT as a reference.

Methods

This retrospective analysis included 381 patients with 1,799 lymph nodes (450 malignant, 1,349 negative). The data set was divided into a training and validation set. A radiomics analysis with 4 filters and 6 algorithms resulting in 24 different radiomics signatures and a bootstrap algorithm (Bagging) with 30 bootstrap iterations was performed. A decision curve analysis was applied to generate a net benefit to compare the radiomics signature to two expert radiologists as one-by-one and as a prescreening tool in combination with the respective radiologist and only the radiologists.

Results

All 24 modeling methods showed good and reliable discrimination for malignant/benign LNs (area under the curve 0.75−0.87). The decision curve analysis showed a net benefit for the least absolute shrinkage and selection operator (LASSO) classifier for the entire probability range and outperformed the expert radiologists except for the high probability range. Using the radiomics signature as a prescreening tool for the radiologists did not improve net benefit.

Conclusions

Radiomics showed good discrimination power irrespective of the modeling technique in detecting LN metastases in patients with known lung cancer. The LASSO classifier was a suitable diagnostic tool and even outperformed the expert radiologists, except for high probabilities. Radiomics failed to improve clinical benefit as a prescreening tool.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41747-022-00296-8.

Intra-scanner repeatability of quantitative imaging features in a 3D printed semi-anthropomorphic CT phantom

OBJECTIVES

Radiomics has shown to provide novel diagnostic and predictive disease information based on quantitative image features in study settings. However, limited data yielded contradictory results and important questions regarding the validity of the methods remain to be answered. The purpose of this study was to evaluate how clinical imaging techniques affect the stability of radiomics features by using 3D printed anthropomorphic CT phantom to test for repeatability and reproducibility of quantitative parameters.

METHODS

48 PET/CT validated lymph nodes of prostate cancer patients (24 metastatic, 24 non-metastatic) were used as a template to create a customized 3D printed anthropomorphic phantom. We subsequently scanned the phantom five times with a routine abdominal CT protocol. Images were reconstructed using iterative reconstruction and two soft tissue kernels and one bone kernel. Radiomics features were extracted and assessed for repeatability and susceptibility towards image reconstruction settings using concordance correlation coefficients.

RESULTS

Our analysis revealed 19 of 86 features (22 %) as highly repeatable (CCC ≥ 0.85) with low susceptibility towards image reconstruction protocols. Most features analyzed depicted critical non-repeatability with CCC's < 0.75 even under entirely consistent imaging acquisition settings. Edge enhancing kernels result in higher variances between the scans and differences in repeatability and reproducibility were detected between PSMA-positive and negative lymph nodes with overall more stable features seen in tumor positive lymph nodes.

CONCLUSIONS

Both, repeatability and reproducibility play a crucial role in the validation process of radiomics features in clinical routine. This phantom study shows that most radiomics features in contrast to previous studies, including phantom and clinical, do not depict sufficient intra-scanner repeatability to serve as reliable diagnostic tools.

Automatized Hepatic Tumor Volume Analysis of Neuroendocrine Liver Metastases by Gd-EOB MRI-A Deep-Learning Model to Support Multidisciplinary Cancer Conference Decision-Making

BACKGROUND

Rapid quantification of liver metastasis for diagnosis and follow-up is an unmet medical need in patients with secondary liver malignancies. We present a 3D-quantification model of neuroendocrine liver metastases (NELM) using gadoxetic-acid (Gd-EOB)-enhanced MRI as a useful tool for multidisciplinary cancer conferences (MCC).

METHODS

Manual 3D-segmentations of NELM and livers (149 patients in 278 Gd-EOB MRI scans) were used to train a neural network (U-Net architecture). Clinical usefulness was evaluated in another 33 patients who were discussed in our MCC and received a Gd-EOB MRI both at baseline and follow-up examination (n = 66) over 12 months. Model measurements (NELM volume; hepatic tumor load (HTL)) with corresponding absolute (ΔabsNELM; ΔabsHTL) and relative changes (ΔrelNELM; ΔrelHTL) between baseline and follow-up were compared to MCC decisions (therapy success/failure).

RESULTS

Internal validation of the model's accuracy showed a high overlap for NELM and livers (Matthew's correlation coefficient (φ): 0.76/0.95, respectively) with higher φ in larger NELM volume (φ = 0.80 vs. 0.71; p = 0.003). External validation confirmed the high accuracy for NELM (φ = 0.86) and livers (φ = 0.96). MCC decisions were significantly differentiated by all response variables (ΔabsNELM; ΔabsHTL; ΔrelNELM; ΔrelHTL) (p < 0.001). ΔrelNELM and ΔrelHTL showed optimal discrimination between therapy success or failure (AUC: 1.000; p < 0.001).

CONCLUSION

The model shows high accuracy in 3D-quantification of NELM and HTL in Gd-EOB-MRI. The model's measurements correlated well with MCC's evaluation of therapeutic response.

Optimizing size thresholds for detection of clinically significant prostate cancer on MRI: Peripheral zone cancers are smaller and more predictable than transition zone tumors

PURPOSE

To evaluate if size-based cut-offs based on MR imaging can successfully assess clinically significant prostate cancer (csPCA). The goal was to improve the currently applied size-based differentiation criterion in PI-RADS.

METHODS AND MATERIALS

MRIs of 293 patients who had undergone 3 T MR imaging with subsequent confirmation of prostate cancer on systematic and targeted MRI/TRUS-fusion biopsy were re-read by three radiologists. All identifiable tumors were measured on T2WI for lesions originating in the transition zone (TZ) and on DWI for lesions from the peripheral zone (PZ) and tabulated against their Gleason grade.

RESULTS

309 lesions were analyzed, 213 (68.9 %) in the PZ and 96 (31.1 %) in the TZ. ROC-Analysis showed a stronger correlation between lesion size and clinically significant (defined as Gleason Grade Group ≥ 2) prostate cancer (PCa) for the PZ (AUC = 0.73) compared to the TZ (AUC = 0.63). The calculated Youden index resulted in size cut-offs of 14 mm for PZ and 21 mm for TZ tumors.

CONCLUSION

Size cut-offs can be used to stratify prostate cancer with different optimal size thresholds in the peripheral zone and transition zone. There was a clearer separation of clinically significant tumors in peripheral zone cancers compared to transition zone cancers. Future iterations of PI-RADS could therefore take different size-based cut-offs for peripheral zone and transition zone cancers into account.

Validation of the PI-RADS language: predictive values of PI-RADS lexicon descriptors for detection of prostate cancer

OBJECTIVES

To assess the discriminatory power of lexicon terms used in PI-RADS version 2 to describe MRI features of prostate lesions.

METHODS

Four hundred fifty-four patients were included in this retrospective, institutional review board-approved study. Patients received multiparametric (mp) MRI and subsequent prostate biopsy including MRI/transrectal ultrasound fusion biopsy and 10-core systematic biopsy. PI-RADS lexicon terms describing lesion characteristics on mpMRI were assigned to lesions by experienced readers. Positive and negative predictive values (PPV, NPV) of each lexicon term were assessed using biopsy results as a reference standard.

RESULTS

From a total of 501 lesions, clinically significant prostate cancer (csPCa) was present in 175 lesions (34.9%). Terms related to findings of restricted diffusion showed PPVs of up to 52.0%/43.9% and NPV of up to 91.8%/89.7% (peripheral zone or PZ/transition zone or TZ). T2-weighted imaging (T2W)-related terms showed a wide range of predictive values. For PZ lesions, high PPVs were found for "markedly hypointense," "lenticular," "lobulated," and "spiculated" (PPVs between 67.2 and 56.7%). For TZ lesions, high PPVs were found for "water-drop-shaped" and "erased charcoal sign" (78.6% and 61.0%). The terms "encapsulated," "organized chaos," and "linear" showed to be good predictors for benignity with distinctively low PPVs between 5.4 and 6.9%. Most T2WI-related terms showed improved predictive values for TZ lesions when combined with DWI-related findings.

CONCLUSIONS

Lexicon terms with high discriminatory power were identified (e.g., "markedly hypointense," "water-drop-shaped," "organized chaos"). DWI-related terms can be useful for excluding TZ cancer. Combining T2WI- with DWI findings in TZ lesions markedly improved predictive values.

KEY POINTS

• Lexicon terms describing morphological and functional features of prostate lesions on MRI show a wide range of predictive values for prostate cancer. • Some T2-related terms have favorable PPVs, e.g., "water-drop-shaped" and "organized chaos" while others show less distinctive predictive values. DWI-related terms have noticeable negative predictive values in TZ lesions making DWI feature a useful tool for exclusion of TZ cancer. • Combining DWI- and T2-related lexicon terms for assessment of TZ lesions markedly improves PPVs. Most T2-related lexicon terms showed a significant decrease in PPV when combined with negative findings for "DW hyperintensity."

Prostate Cancer Nodal Staging: Using Deep Learning to Predict 68Ga-PSMA-Positivity from CT Imaging Alone

Lymphatic spread determines treatment decisions in prostate cancer (PCa) patients. 68Ga-PSMA-PET/CT can be performed, although cost remains high and availability is limited. Therefore, computed tomography (CT) continues to be the most used modality for PCa staging. We assessed if convolutional neural networks (CNNs) can be trained to determine 68Ga-PSMA-PET/CT-lymph node status from CT alone. In 549 patients with 68Ga-PSMA PET/CT imaging, 2616 lymph nodes were segmented. Using PET as a reference standard, three CNNs were trained. Training sets balanced for infiltration status, lymph node location and additionally, masked images, were used for training. CNNs were evaluated using a separate test set and performance was compared to radiologists' assessments and random forest classifiers. Heatmaps maps were used to identify the performance determining image regions. The CNNs performed with an Area-Under-the-Curve of 0.95 (status balanced) and 0.86 (location balanced, masked), compared to an AUC of 0.81 of experienced radiologists. Interestingly, CNNs used anatomical surroundings to increase their performance, "learning" the infiltration probabilities of anatomical locations. In conclusion, CNNs have the potential to build a well performing CT-based biomarker for lymph node metastases in PCa, with different types of class balancing strongly affecting CNN performance.