Using histopathology latent diffusion models as privacy-preserving dataset augmenters improves downstream classification performance

Latent diffusion models (LDMs) have emerged as a state-of-the-art image generation method, outperforming previous Generative Adversarial Networks (GANs) in terms of training stability and image quality. In computational pathology, generative models are valuable for data sharing and data augmentation. However, the impact of LDM-generated images on histopathology tasks compared to traditional GANs has not been systematically studied. We trained three LDMs and a styleGAN2 model on histology tiles from nine colorectal cancer (CRC) tissue classes. The LDMs include 1) a fine-tuned version of stable diffusion v1.4, 2) a Kullback-Leibler (KL)-autoencoder (KLF8-DM), and 3) a vector quantized (VQ)-autoencoder deploying LDM (VQF8-DM). We assessed image quality through expert ratings, dimensional reduction methods, distribution similarity measures, and their impact on training a multiclass tissue classifier. Additionally, we investigated image memorization in the KLF8-DM and styleGAN2 models. All models provided a high image quality, with the KLF8-DM achieving the best Frechet Inception Distance (FID) and expert rating scores for complex tissue classes. For simpler classes, the VQF8-DM and styleGAN2 models performed better. Image memorization was negligible for both styleGAN2 and KLF8-DM models. Classifiers trained on a mix of KLF8-DM generated and real images achieved a 4% improvement in overall classification accuracy, highlighting the usefulness of these images for dataset augmentation. Our systematic study of generative methods showed that KLF8-DM produces the highest quality images with negligible image memorization. The higher classifier performance in the generatively augmented dataset suggests that this augmentation technique can be employed to enhance histopathology classifiers for various tasks.

Preoperative three-dimensional lung volumetry predicts respiratory complications in patients undergoing major liver resection for colorectal metastases

Colorectal liver metastases (CRLM) are the predominant factor limiting survival in patients with colorectal cancer and liver resection with complete tumor removal is the best treatment option for these patients. This study examines the predictive ability of three-dimensional lung volumetry (3DLV) based on preoperative computerized tomography (CT), to predict postoperative pulmonary complications in patients undergoing major liver resection for CRLM. Patients undergoing major curative liver resection for CRLM between 2010 and 2021 with a preoperative CT scan of the thorax within 6 weeks of surgery, were included. Total lung volume (TLV) was calculated using volumetry software 3D-Slicer version 4.11.20210226 including Chest Imaging Platform extension ( http://www.slicer.org ). The area under the curve (AUC) of a receiver-operating characteristic analysis was used to define a cut-off value of TLV, for predicting the occurrence of postoperative respiratory complications. Differences between patients with TLV below and above the cut-off were examined with Chi-square or Fisher's exact test and Mann-Whitney U tests and logistic regression was used to determine independent risk factors for the development of respiratory complications. A total of 123 patients were included, of which 35 (29%) developed respiratory complications. A predictive ability of TLV regarding respiratory complications was shown (AUC 0.62, p = 0.036) and a cut-off value of 4500 cm3 was defined. Patients with TLV < 4500 cm3 were shown to suffer from significantly higher rates of respiratory complications (44% vs. 21%, p = 0.007) compared to the rest. Logistic regression analysis identified TLV < 4500 cm3 as an independent predictor for the occurrence of respiratory complications (odds ratio 3.777, 95% confidence intervals 1.488-9.588, p = 0.005). Preoperative 3DLV is a viable technique for prediction of postoperative pulmonary complications in patients undergoing major liver resection for CRLM. More studies in larger cohorts are necessary to further evaluate this technique.

Integrating Text and Image Analysis: Exploring GPT-4V's Capabilities in Advanced Radiological Applications Across Subspecialties

This study demonstrates that GPT-4V outperforms GPT-4 across radiology subspecialties in analyzing 207 cases with 1312 images from the Radiological Society of North America Case Collection.

Diffusion probabilistic versus generative adversarial models to reduce contrast agent dose in breast MRI

BACKGROUND

To compare denoising diffusion probabilistic models (DDPM) and generative adversarial networks (GAN) for recovering contrast-enhanced breast magnetic resonance imaging (MRI) subtraction images from virtual low-dose subtraction images.

METHODS

Retrospective, ethically approved study. DDPM- and GAN-reconstructed single-slice subtraction images of 50 breasts with enhancing lesions were compared to original ones at three dose levels (25%, 10%, 5%) using quantitative measures and radiologic evaluations. Two radiologists stated their preference based on the reconstruction quality and scored the lesion conspicuity as compared to the original, blinded to the model. Fifty lesion-free maximum intensity projections were evaluated for the presence of false-positives. Results were compared between models and dose levels, using generalized linear mixed models.

RESULTS

At 5% dose, both radiologists preferred the GAN-generated images, whereas at 25% dose, both radiologists preferred the DDPM-generated images. Median lesion conspicuity scores did not differ between GAN and DDPM at 25% dose (5 versus 5, p = 1.000) and 10% dose (4 versus 4, p = 1.000). At 5% dose, both readers assigned higher conspicuity to the GAN than to the DDPM (3 versus 2, p = 0.007). In the lesion-free examinations, DDPM and GAN showed no differences in the false-positive rate at 5% (15% versus 22%), 10% (10% versus 6%), and 25% (6% versus 4%) (p = 1.000).

CONCLUSIONS

Both GAN and DDPM yielded promising results in low-dose image reconstruction. However, neither of them showed superior results over the other model for all dose levels and evaluation metrics. Further development is needed to counteract false-positives.

RELEVANCE STATEMENT

For MRI-based breast cancer screening, reducing the contrast agent dose is desirable. Diffusion probabilistic models and generative adversarial networks were capable of retrospectively enhancing the signal of low-dose images. Hence, they may supplement imaging with reduced doses in the future.

KEY POINTS

• Deep learning may help recover signal in low-dose contrast-enhanced breast MRI. • Two models (DDPM and GAN) were trained at different dose levels. • Radiologists preferred DDPM at 25%, and GAN images at 5% dose. • Lesion conspicuity between DDPM and GAN was similar, except at 5% dose. • GAN and DDPM yield promising results in low-dose image reconstruction.

Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette Questions

This study compares 2 large language models and their performance vs that of competing open-source models.

The virtual reference radiologist: comprehensive AI assistance for clinical image reading and interpretation

OBJECTIVES

Large language models (LLMs) have shown potential in radiology, but their ability to aid radiologists in interpreting imaging studies remains unexplored. We investigated the effects of a state-of-the-art LLM (GPT-4) on the radiologists' diagnostic workflow.

MATERIALS AND METHODS

In this retrospective study, six radiologists of different experience levels read 40 selected radiographic [n = 10], CT [n = 10], MRI [n = 10], and angiographic [n = 10] studies unassisted (session one) and assisted by GPT-4 (session two). Each imaging study was presented with demographic data, the chief complaint, and associated symptoms, and diagnoses were registered using an online survey tool. The impact of Artificial Intelligence (AI) on diagnostic accuracy, confidence, user experience, input prompts, and generated responses was assessed. False information was registered. Linear mixed-effect models were used to quantify the factors (fixed: experience, modality, AI assistance; random: radiologist) influencing diagnostic accuracy and confidence.

RESULTS

When assessing if the correct diagnosis was among the top-3 differential diagnoses, diagnostic accuracy improved slightly from 181/240 (75.4%, unassisted) to 188/240 (78.3%, AI-assisted). Similar improvements were found when only the top differential diagnosis was considered. AI assistance was used in 77.5% of the readings. Three hundred nine prompts were generated, primarily involving differential diagnoses (59.1%) and imaging features of specific conditions (27.5%). Diagnostic confidence was significantly higher when readings were AI-assisted (p > 0.001). Twenty-three responses (7.4%) were classified as hallucinations, while two (0.6%) were misinterpretations.

CONCLUSION

Integrating GPT-4 in the diagnostic process improved diagnostic accuracy slightly and diagnostic confidence significantly. Potentially harmful hallucinations and misinterpretations call for caution and highlight the need for further safeguarding measures.

CLINICAL RELEVANCE STATEMENT

Using GPT-4 as a virtual assistant when reading images made six radiologists of different experience levels feel more confident and provide more accurate diagnoses; yet, GPT-4 gave factually incorrect and potentially harmful information in 7.4% of its responses.

Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis

BACKGROUND

The field of Artificial Intelligence (AI) holds transformative potential in medicine. However, the lack of universal reporting guidelines poses challenges in ensuring the validity and reproducibility of published research studies in this field.

METHODS

Based on a systematic review of academic publications and reporting standards demanded by both international consortia and regulatory stakeholders as well as leading journals in the fields of medicine and medical informatics, 26 reporting guidelines published between 2009 and 2023 were included in this analysis. Guidelines were stratified by breadth (general or specific to medical fields), underlying consensus quality, and target research phase (preclinical, translational, clinical) and subsequently analyzed regarding the overlap and variations in guideline items.

RESULTS

AI reporting guidelines for medical research vary with respect to the quality of the underlying consensus process, breadth, and target research phase. Some guideline items such as reporting of study design and model performance recur across guidelines, whereas other items are specific to particular fields and research stages.

CONCLUSIONS

Our analysis highlights the importance of reporting guidelines in clinical AI research and underscores the need for common standards that address the identified variations and gaps in current guidelines. Overall, this comprehensive overview could help researchers and public stakeholders reinforce quality standards for increased reliability, reproducibility, clinical validity, and public trust in AI research in healthcare. This could facilitate the safe, effective, and ethical translation of AI methods into clinical applications that will ultimately improve patient outcomes.

[Current MR imaging of cartilage in the context of knee osteoarthritis (part 2) : Cartilage pathologies and their assessment]

High-quality magnetic resonance (MR) imaging is essential for the precise assessment of the knee joint and plays a key role in the diagnostics, treatment and prognosis. Intact cartilage tissue is characterized by a smooth surface, uniform tissue thickness and an organized zonal structure, which are manifested as depth-dependent signal intensity variations. Cartilage pathologies are identifiable through alterations in signal intensity and morphology and should be communicated based on a precise terminology. Cartilage pathologies can show hyperintense and hypointense signal alterations. Cartilage defects are assessed based on their depth and should be described in terms of their location and extent. The following symptom constellations are of overarching clinical relevance in image reading and interpretation: symptom constellations associated with rapidly progressive forms of joint degeneration and unfavorable prognosis, accompanying symptom constellations mostly in connection with destabilizing meniscal lesions and subchondral insufficiency fractures (accelerated osteoarthritis) as well as symptoms beyond the "typical" degeneration, especially when a discrepancy is observed between (minor) structural changes and (major) synovitis and effusion (inflammatory arthropathy).

Will I soon be out of my job? Quality and guideline conformity of ChatGPT therapy suggestions to patient inquiries with gynecologic symptoms in a palliative setting

PURPOSE

The market and application possibilities for artificial intelligence are currently growing at high speed and are increasingly finding their way into gynecology. While the medical side is highly represented in the current literature, the patient's perspective is still lagging behind. Therefore, the aim of this study was to evaluate the recommendations of ChatGPT regarding patient inquiries about the possible therapy of gynecological leading symptoms in a palliative situation by experts.

METHODS

Case vignettes were constructed for 10 common concomitant symptoms in gynecologic oncology tumors in a palliative setting, and patient queries regarding therapy of these symptoms were generated as prompts for ChatGPT. Five experts in palliative care and gynecologic oncology evaluated the responses with respect to guideline adherence and applicability and identified advantages and disadvantages.

RESULTS

The overall rating of ChatGPT responses averaged 4.1 (5 = strongly agree; 1 = strongly disagree). The experts saw an average guideline conformity of the therapy recommendations with a value of 4.0. ChatGPT sometimes omits relevant therapies and does not provide an individual assessment of the suggested therapies, but does indicate that a physician consultation is additionally necessary.

CONCLUSIONS

Language models, such as ChatGPT, can provide valid and largely guideline-compliant therapy recommendations in their freely available and thus in principle accessible version for our patients. For a complete therapy recommendation, an evaluation of the therapies, their individual adjustment as well as a filtering of possible wrong recommendations, a medical expert's opinion remains indispensable.

[Current MR imaging of cartilage in the context of knee osteoarthritis (part 1) : Principles and sequences]

Magnetic resonance imaging (MRI) is the clinical method of choice for cartilage imaging in the context of degenerative and nondegenerative joint diseases. The MRI-based definitions of osteoarthritis rely on the detection of osteophytes, cartilage pathologies, bone marrow edema and meniscal lesions but currently a scientific consensus is lacking. In the clinical routine proton density-weighted, fat-suppressed 2D turbo spin echo sequences with echo times of 30-40 ms are predominantly used, which are sufficiently sensitive and specific for the assessment of cartilage. The additionally acquired T1-weighted sequences are primarily used for evaluating other intra-articular and periarticular structures. Diagnostically relevant artifacts include magic angle and chemical shift artifacts, which can lead to artificial signal enhancement in cartilage or incorrect representations of the subchondral lamina and its thickness. Although scientifically validated, high-resolution 3D gradient echo sequences (for cartilage segmentation) and compositional MR sequences (for quantification of physical tissue parameters) are currently reserved for scientific research questions. The future integration of artificial intelligence techniques in areas such as image reconstruction (to reduce scan times while maintaining image quality), image analysis (for automated identification of cartilage defects), and image postprocessing (for automated segmentation of cartilage in terms of volume and thickness) will significantly improve the diagnostic workflow and advance the field further.

Large language models and multimodal foundation models for precision oncology

The technological progress in artificial intelligence (AI) has massively accelerated since 2022, with far-reaching implications for oncology and cancer research. Large language models (LLMs) now perform at human-level competency in text processing. Notably, both text and image processing networks are increasingly based on transformer neural networks. This convergence enables the development of multimodal AI models that take diverse types of data as an input simultaneously, marking a qualitative shift from specialized niche models which were prevalent in the 2010s. This editorial summarizes these developments, which are expected to impact precision oncology in the coming years.

Preserving fairness and diagnostic accuracy in private large-scale AI models for medical imaging

BACKGROUND

Artificial intelligence (AI) models are increasingly used in the medical domain. However, as medical data is highly sensitive, special precautions to ensure its protection are required. The gold standard for privacy preservation is the introduction of differential privacy (DP) to model training. Prior work indicates that DP has negative implications on model accuracy and fairness, which are unacceptable in medicine and represent a main barrier to the widespread use of privacy-preserving techniques. In this work, we evaluated the effect of privacy-preserving training of AI models regarding accuracy and fairness compared to non-private training.

METHODS

We used two datasets: (1) A large dataset (N = 193,311) of high quality clinical chest radiographs, and (2) a dataset (N = 1625) of 3D abdominal computed tomography (CT) images, with the task of classifying the presence of pancreatic ductal adenocarcinoma (PDAC). Both were retrospectively collected and manually labeled by experienced radiologists. We then compared non-private deep convolutional neural networks (CNNs) and privacy-preserving (DP) models with respect to privacy-utility trade-offs measured as area under the receiver operating characteristic curve (AUROC), and privacy-fairness trade-offs, measured as Pearson's r or Statistical Parity Difference.

RESULTS

We find that, while the privacy-preserving training yields lower accuracy, it largely does not amplify discrimination against age, sex or co-morbidity. However, we find an indication that difficult diagnoses and subgroups suffer stronger performance hits in private training.

CONCLUSIONS

Our study shows that - under the challenging realistic circumstances of a real-life clinical dataset - the privacy-preserving training of diagnostic deep learning models is possible with excellent diagnostic accuracy and fairness.

International pharmacy students' perceptions towards artificial intelligence in medicine-A multinational, multicentre cross-sectional study

AIMS

To explore international undergraduate pharmacy students' views on integrating artificial intelligence (AI) into pharmacy education and practice.

METHODS

This cross-sectional institutional review board-approved multinational, multicentre study comprised an anonymous online survey of 14 multiple-choice items to assess pharmacy students' preferences for AI events in the pharmacy curriculum, the current state of AI education, and students' AI knowledge and attitudes towards using AI in the pharmacy profession, supplemented by 8 demographic queries. Subgroup analyses were performed considering sex, study year, tech-savviness, and prior AI knowledge and AI events in the curriculum using the Mann-Whitney U-test. Variances were reported for responses in Likert scale format.

RESULTS

The survey gathered 387 pharmacy student opinions across 16 faculties and 12 countries. Students showed predominantly positive attitudes towards AI in medicine (58%, n = 225) and expressed a strong desire for more AI education (72%, n = 276). However, they reported limited general knowledge of AI (63%, n = 242) and felt inadequately prepared to use AI in their future careers (51%, n = 197). Male students showed more positive attitudes towards increasing efficiency through AI (P = .011), while tech-savvy and advanced-year students expressed heightened concerns about potential legal and ethical issues related to AI (P < .001/P = .025, respectively). Students who had AI courses as part of their studies reported better AI knowledge (P < .001) and felt more prepared to apply it professionally (P < .001).

CONCLUSIONS

Our findings underline the generally positive attitude of international pharmacy students towards AI application in medicine and highlight the necessity for a greater emphasis on AI education within pharmacy curricula.

Extracting structured information from unstructured histopathology reports using generative pre-trained transformer 4 (GPT-4)

Deep learning applied to whole-slide histopathology images (WSIs) has the potential to enhance precision oncology and alleviate the workload of experts. However, developing these models necessitates large amounts of data with ground truth labels, which can be both time-consuming and expensive to obtain. Pathology reports are typically unstructured or poorly structured texts, and efforts to implement structured reporting templates have been unsuccessful, as these efforts lead to perceived extra workload. In this study, we hypothesised that large language models (LLMs), such as the generative pre-trained transformer 4 (GPT-4), can extract structured data from unstructured plain language reports using a zero-shot approach without requiring any re-training. We tested this hypothesis by utilising GPT-4 to extract information from histopathological reports, focusing on two extensive sets of pathology reports for colorectal cancer and glioblastoma. We found a high concordance between LLM-generated structured data and human-generated structured data. Consequently, LLMs could potentially be employed routinely to extract ground truth data for machine learning from unstructured pathology reports in the future. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Evaluation of Pulmonary Nodules by Radiologists vs. Radiomics in Stand-Alone and Complementary CT and MRI

Increased attention has been given to MRI in radiation-free screening for malignant nodules in recent years. Our objective was to compare the performance of human readers and radiomic feature analysis based on stand-alone and complementary CT and MRI imaging in classifying pulmonary nodules. This single-center study comprises patients with CT findings of pulmonary nodules who underwent additional lung MRI and whose nodules were classified as benign/malignant by resection. For radiomic features analysis, 2D segmentation was performed for each lung nodule on axial CT, T2-weighted (T2w), and diffusion (DWI) images. The 105 extracted features were reduced by iterative backward selection. The performance of radiomics and human readers was compared by calculating accuracy with Clopper-Pearson confidence intervals. Fifty patients (mean age 63 +/- 10 years) with 66 pulmonary nodules (40 malignant) were evaluated. ACC values for radiomic features analysis vs. radiologists based on CT alone (0.68; 95%CI: 0.56, 0.79 vs. 0.59; 95%CI: 0.46, 0.71), T2w alone (0.65; 95%CI: 0.52, 0.77 vs. 0.68; 95%CI: 0.54, 0.78), DWI alone (0.61; 95%CI:0.48, 0.72 vs. 0.73; 95%CI: 0.60, 0.83), combined T2w/DWI (0.73; 95%CI: 0.60, 0.83 vs. 0.70; 95%CI: 0.57, 0.80), and combined CT/T2w/DWI (0.83; 95%CI: 0.72, 0.91 vs. 0.64; 95%CI: 0.51, 0.75) were calculated. This study is the first to show that by combining quantitative image information from CT, T2w, and DWI datasets, pulmonary nodule assessment through radiomics analysis is superior to using one modality alone, even exceeding human readers' performance.

Large language models streamline automated machine learning for clinical studies

A knowledge gap persists between machine learning (ML) developers (e.g., data scientists) and practitioners (e.g., clinicians), hampering the full utilization of ML for clinical data analysis. We investigated the potential of the ChatGPT Advanced Data Analysis (ADA), an extension of GPT-4, to bridge this gap and perform ML analyses efficiently. Real-world clinical datasets and study details from large trials across various medical specialties were presented to ChatGPT ADA without specific guidance. ChatGPT ADA autonomously developed state-of-the-art ML models based on the original study's training data to predict clinical outcomes such as cancer development, cancer progression, disease complications, or biomarkers such as pathogenic gene sequences. Following the re-implementation and optimization of the published models, the head-to-head comparison of the ChatGPT ADA-crafted ML models and their respective manually crafted counterparts revealed no significant differences in traditional performance metrics (p ≥ 0.072). Strikingly, the ChatGPT ADA-crafted ML models often outperformed their counterparts. In conclusion, ChatGPT ADA offers a promising avenue to democratize ML in medicine by simplifying complex data analyses, yet should enhance, not replace, specialized training and resources, to promote broader applications in medical research and practice.

Enhancing diagnostic deep learning via self-supervised pretraining on large-scale, unlabeled non-medical images

BACKGROUND

Pretraining labeled datasets, like ImageNet, have become a technical standard in advanced medical image analysis. However, the emergence of self-supervised learning (SSL), which leverages unlabeled data to learn robust features, presents an opportunity to bypass the intensive labeling process. In this study, we explored if SSL for pretraining on non-medical images can be applied to chest radiographs and how it compares to supervised pretraining on non-medical images and on medical images.

METHODS

We utilized a vision transformer and initialized its weights based on the following: (i) SSL pretraining on non-medical images (DINOv2), (ii) supervised learning (SL) pretraining on non-medical images (ImageNet dataset), and (iii) SL pretraining on chest radiographs from the MIMIC-CXR database, the largest labeled public dataset of chest radiographs to date. We tested our approach on over 800,000 chest radiographs from 6 large global datasets, diagnosing more than 20 different imaging findings. Performance was quantified using the area under the receiver operating characteristic curve and evaluated for statistical significance using bootstrapping.

RESULTS

SSL pretraining on non-medical images not only outperformed ImageNet-based pretraining (p < 0.001 for all datasets) but, in certain cases, also exceeded SL on the MIMIC-CXR dataset. Our findings suggest that selecting the right pretraining strategy, especially with SSL, can be pivotal for improving diagnostic accuracy of artificial intelligence in medical imaging.

CONCLUSIONS

By demonstrating the promise of SSL in chest radiograph analysis, we underline a transformative shift towards more efficient and accurate AI models in medical imaging.

RELEVANCE STATEMENT

Self-supervised learning highlights a paradigm shift towards the enhancement of AI-driven accuracy and efficiency in medical imaging. Given its promise, the broader application of self-supervised learning in medical imaging calls for deeper exploration, particularly in contexts where comprehensive annotated datasets are limited.

Wearable activity data can predict functional recovery after musculoskeletal injury: Feasibility of a machine learning approach

Delayed functional recovery after injury is associated with significant personal and socioeconomic burden. Identification of patients at risk for a prolonged recovery after a musculoskeletal injury is thus of high relevance. The aim of the current study was to show the feasibility of using a machine learning assisted model to predict functional recovery based on the pre- and immediate post injury patient activity as measured with wearable systems in trauma patients. Patients with a pre-existing wearable (smartphone and/or body-worn sensor), data availability of at least 7 days prior to their injury, and any musculoskeletal injury of the upper or lower extremity were included in this study. Patient age, sex, injured extremity, time off work and step count as activity data were recorded continuously both pre- and post-injury. Descriptive statistics were performed and a logistic regression machine learning model was used to predict the patient's functional recovery status after 6 weeks based on their pre- and post-injury activity characteristics. Overall 38 patients (7 upper extremity, 24 lower extremity, 5 pelvis, 2 combined) were included in this proof-of-concept study. The average follow-up with available wearable data was 85.4 days. Based on the activity data, a predictive model was constructed to determine the likelihood of having a recovery of at least 50 % of the pre-injury activity state by post injury week 6. Based on the individual activity by week 3 a predictive accuracy of over 80 % was achieved on an independent test set (F1=0,82; AUC=0,86; ACC=8,83). The employed model is feasible to assess the principal risk for a slower recovery based on readily available personal wearable activity data. The model has the potential to identify patients requiring additional aftercare attention early during the treatment course, thus optimizing return to the pre-injury status through focused interventions. Additional patient data is needed to adapt the model to more specifically focus on different fracture entities and patient groups.

Encrypted federated learning for secure decentralized collaboration in cancer image analysis

Artificial intelligence (AI) has a multitude of applications in cancer research and oncology. However, the training of AI systems is impeded by the limited availability of large datasets due to data protection requirements and other regulatory obstacles. Federated and swarm learning represent possible solutions to this problem by collaboratively training AI models while avoiding data transfer. However, in these decentralized methods, weight updates are still transferred to the aggregation server for merging the models. This leaves the possibility for a breach of data privacy, for example by model inversion or membership inference attacks by untrusted servers. Somewhat-homomorphically-encrypted federated learning (SHEFL) is a solution to this problem because only encrypted weights are transferred, and model updates are performed in the encrypted space. Here, we demonstrate the first successful implementation of SHEFL in a range of clinically relevant tasks in cancer image analysis on multicentric datasets in radiology and histopathology. We show that SHEFL enables the training of AI models which outperform locally trained models and perform on par with models which are centrally trained. In the future, SHEFL can enable multiple institutions to co-train AI models without forsaking data governance and without ever transmitting any decryptable data to untrusted servers.

End-to-end prognostication in colorectal cancer by deep learning: a retrospective, multicentre study

BACKGROUND

Precise prognosis prediction in patients with colorectal cancer (ie, forecasting survival) is pivotal for individualised treatment and care. Histopathological tissue slides of colorectal cancer specimens contain rich prognostically relevant information. However, existing studies do not have multicentre external validation with real-world sample processing protocols, and algorithms are not yet widely used in clinical routine.

METHODS

In this retrospective, multicentre study, we collected tissue samples from four groups of patients with resected colorectal cancer from Australia, Germany, and the USA. We developed and externally validated a deep learning-based prognostic-stratification system for automatic prediction of overall and cancer-specific survival in patients with resected colorectal cancer. We used the model-predicted risk scores to stratify patients into different risk groups and compared survival outcomes between these groups. Additionally, we evaluated the prognostic value of these risk groups after adjusting for established prognostic variables.

FINDINGS

We trained and validated our model on a total of 4428 patients. We found that patients could be divided into high-risk and low-risk groups on the basis of the deep learning-based risk score. On the internal test set, the group with a high-risk score had a worse prognosis than the group with a low-risk score, as reflected by a hazard ratio (HR) of 4·50 (95% CI 3·33-6·09) for overall survival and 8·35 (5·06-13·78) for disease-specific survival (DSS). We found consistent performance across three large external test sets. In a test set of 1395 patients, the high-risk group had a lower DSS than the low-risk group, with an HR of 3·08 (2·44-3·89). In two additional test sets, the HRs for DSS were 2·23 (1·23-4·04) and 3·07 (1·78-5·3). We showed that the prognostic value of the deep learning-based risk score is independent of established clinical risk factors.

INTERPRETATION

Our findings indicate that attention-based self-supervised deep learning can robustly offer a prognosis on clinical outcomes in patients with colorectal cancer, generalising across different populations and serving as a potentially new prognostic tool in clinical decision making for colorectal cancer management. We release all source codes and trained models under an open-source licence, allowing other researchers to reuse and build upon our work.

FUNDING

The German Federal Ministry of Health, the Max-Eder-Programme of German Cancer Aid, the German Federal Ministry of Education and Research, the German Academic Exchange Service, and the EU.

AIROGS: Artificial Intelligence for Robust Glaucoma Screening Challenge

The early detection of glaucoma is essential in preventing visual impairment. Artificial intelligence (AI) can be used to analyze color fundus photographs (CFPs) in a cost-effective manner, making glaucoma screening more accessible. While AI models for glaucoma screening from CFPs have shown promising results in laboratory settings, their performance decreases significantly in real-world scenarios due to the presence of out-of-distribution and low-quality images. To address this issue, we propose the Artificial Intelligence for Robust Glaucoma Screening (AIROGS) challenge. This challenge includes a large dataset of around 113,000 images from about 60,000 patients and 500 different screening centers, and encourages the development of algorithms that are robust to ungradable and unexpected input data. We evaluated solutions from 14 teams in this paper and found that the best teams performed similarly to a set of 20 expert ophthalmologists and optometrists. The highest-scoring team achieved an area under the receiver operating characteristic curve of 0.99 (95% CI: 0.98-0.99) for detecting ungradable images on-the-fly. Additionally, many of the algorithms showed robust performance when tested on three other publicly available datasets. These results demonstrate the feasibility of robust AI-enabled glaucoma screening.

Enhancing domain generalization in the AI-based analysis of chest radiographs with federated learning

Developing robust artificial intelligence (AI) models that generalize well to unseen datasets is challenging and usually requires large and variable datasets, preferably from multiple institutions. In federated learning (FL), a model is trained collaboratively at numerous sites that hold local datasets without exchanging them. So far, the impact of training strategy, i.e., local versus collaborative, on the diagnostic on-domain and off-domain performance of AI models interpreting chest radiographs has not been assessed. Consequently, using 610,000 chest radiographs from five institutions across the globe, we assessed diagnostic performance as a function of training strategy (i.e., local vs. collaborative), network architecture (i.e., convolutional vs. transformer-based), single versus cross-institutional performance (i.e., on-domain vs. off-domain), imaging finding (i.e., cardiomegaly, pleural effusion, pneumonia, atelectasis, consolidation, pneumothorax, and no abnormality), dataset size (i.e., from n = 18,000 to 213,921 radiographs), and dataset diversity. Large datasets not only showed minimal performance gains with FL but, in some instances, even exhibited decreases. In contrast, smaller datasets revealed marked improvements. Thus, on-domain performance was mainly driven by training data size. However, off-domain performance leaned more on training diversity. When trained collaboratively across diverse external institutions, AI models consistently surpassed models trained locally for off-domain tasks, emphasizing FL's potential in leveraging data diversity. In conclusion, FL can bolster diagnostic privacy, reproducibility, and off-domain reliability of AI models and, potentially, optimize healthcare outcomes.

A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports

Large language models (LLMs) have shown potential in various applications, including clinical practice. However, their accuracy and utility in providing treatment recommendations for orthopedic conditions remain to be investigated. Thus, this pilot study aims to evaluate the validity of treatment recommendations generated by GPT-4 for common knee and shoulder orthopedic conditions using anonymized clinical MRI reports. A retrospective analysis was conducted using 20 anonymized clinical MRI reports, with varying severity and complexity. Treatment recommendations were elicited from GPT-4 and evaluated by two board-certified specialty-trained senior orthopedic surgeons. Their evaluation focused on semiquantitative gradings of accuracy and clinical utility and potential limitations of the LLM-generated recommendations. GPT-4 provided treatment recommendations for 20 patients (mean age, 50 years ± 19 [standard deviation]; 12 men) with acute and chronic knee and shoulder conditions. The LLM produced largely accurate and clinically useful recommendations. However, limited awareness of a patient's overall situation, a tendency to incorrectly appreciate treatment urgency, and largely schematic and unspecific treatment recommendations were observed and may reduce its clinical usefulness. In conclusion, LLM-based treatment recommendations are largely adequate and not prone to 'hallucinations', yet inadequate in particular situations. Critical guidance by healthcare professionals is obligatory, and independent use by patients is discouraged, given the dependency on precise data input.

Deep learning trained on lymph node status predicts outcome from gastric cancer histopathology: a retrospective multicentric study

AIM

Gastric cancer (GC) is a tumour entity with highly variant outcomes. Lymph node metastasis is a prognostically adverse biomarker. We hypothesised that GC primary tissue contains information that is predictive of lymph node status and patient prognosis and that this information can be extracted using deep learning (DL).

METHODS

Using three patient cohorts comprising 1146 patients, we trained and validated a DL system to predict lymph node status directly from haematoxylin and eosin-stained GC tissue sections. We investigated the concordance between the DL-based prediction from the primary tumour slides (aiN score) and the histopathological lymph node status (pN). Furthermore, we assessed the prognostic value of the aiN score alone and when combined with the pN status.

RESULTS

The aiN score predicted the pN status reaching area under the receiver operating characteristic curves of 0.71 in the training cohort and 0.69 and 0.65 in the two test cohorts. In a multivariate Cox analysis, the aiN score was an independent predictor of patient survival with hazard ratios of 1.5 in the training cohort and of 1.3 and 2.2 in the two test cohorts. A combination of the aiN score and the pN status prognostically stratified patients by survival with p-values <0.05 in logrank tests.

CONCLUSION

GC primary tumour tissue contains additional prognostic information that is accessible using the aiN score. In combination with the pN status, this can be used for personalised management of GC patients after prospective validation.

Automated segmentation of 3D cine cardiovascular magnetic resonance imaging

Introduction

As the life expectancy of children with congenital heart disease (CHD) is rapidly increasing and the adult population with CHD is growing, there is an unmet need to improve clinical workflow and efficiency of analysis. Cardiovascular magnetic resonance (CMR) is a noninvasive imaging modality for monitoring patients with CHD. CMR exam is based on multiple breath-hold 2-dimensional (2D) cine acquisitions that should be precisely prescribed and is expert and institution dependent. Moreover, 2D cine images have relatively thick slices, which does not allow for isotropic delineation of ventricular structures. Thus, development of an isotropic 3D cine acquisition and automatic segmentation method is worthwhile to make CMR workflow straightforward and efficient, as the present work aims to establish.

Methods

Ninety-nine patients with many types of CHD were imaged using a non-angulated 3D cine CMR sequence covering the whole-heart and great vessels. Automatic supervised and semi-supervised deep-learning-based methods were developed for whole-heart segmentation of 3D cine images to separately delineate the cardiac structures, including both atria, both ventricles, aorta, pulmonary arteries, and superior and inferior vena cavae. The segmentation results derived from the two methods were compared with the manual segmentation in terms of Dice score, a degree of overlap agreement, and atrial and ventricular volume measurements.

Results

The semi-supervised method resulted in a better overlap agreement with the manual segmentation than the supervised method for all 8 structures (Dice score 83.23 ± 16.76% vs. 77.98 ± 19.64%; P-value ≤0.001). The mean difference error in atrial and ventricular volumetric measurements between manual segmentation and semi-supervised method was lower (bias ≤ 5.2 ml) than the supervised method (bias ≤ 10.1 ml).

Discussion

The proposed semi-supervised method is capable of cardiac segmentation and chamber volume quantification in a CHD population with wide anatomical variability. It accurately delineates the heart chambers and great vessels and can be used to accurately calculate ventricular and atrial volumes throughout the cardiac cycle. Such a segmentation method can reduce inter- and intra- observer variability and make CMR exams more standardized and efficient.

Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers

Background Clinicians consider both imaging and nonimaging data when diagnosing diseases; however, current machine learning approaches primarily consider data from a single modality. Purpose To develop a neural network architecture capable of integrating multimodal patient data and compare its performance to models incorporating a single modality for diagnosing up to 25 pathologic conditions. Materials and Methods In this retrospective study, imaging and nonimaging patient data were extracted from the Medical Information Mart for Intensive Care (MIMIC) database and an internal database comprised of chest radiographs and clinical parameters inpatients in the intensive care unit (ICU) (January 2008 to December 2020). The MIMIC and internal data sets were each split into training (n = 33 893, n = 28 809), validation (n = 740, n = 7203), and test (n = 1909, n = 9004) sets. A novel transformer-based neural network architecture was trained to diagnose up to 25 conditions using nonimaging data alone, imaging data alone, or multimodal data. Diagnostic performance was assessed using area under the receiver operating characteristic curve (AUC) analysis. Results The MIMIC and internal data sets included 36 542 patients (mean age, 63 years ± 17 [SD]; 20 567 male patients) and 45 016 patients (mean age, 66 years ± 16; 27 577 male patients), respectively. The multimodal model showed improved diagnostic performance for all pathologic conditions. For the MIMIC data set, the mean AUC was 0.77 (95% CI: 0.77, 0.78) when both chest radiographs and clinical parameters were used, compared with 0.70 (95% CI: 0.69, 0.71; P < .001) for only chest radiographs and 0.72 (95% CI: 0.72, 0.73; P < .001) for only clinical parameters. These findings were confirmed on the internal data set. Conclusion A model trained on imaging and nonimaging data outperformed models trained on only one type of data for diagnosing multiple diseases in patients in an ICU setting. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Kitamura and Topol in this issue.

Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study

Deep learning (DL) can accelerate the prediction of prognostic biomarkers from routine pathology slides in colorectal cancer (CRC). However, current approaches rely on convolutional neural networks (CNNs) and have mostly been validated on small patient cohorts. Here, we develop a new transformer-based pipeline for end-to-end biomarker prediction from pathology slides by combining a pre-trained transformer encoder with a transformer network for patch aggregation. Our transformer-based approach substantially improves the performance, generalizability, data efficiency, and interpretability as compared with current state-of-the-art algorithms. After training and evaluating on a large multicenter cohort of over 13,000 patients from 16 colorectal cancer cohorts, we achieve a sensitivity of 0.99 with a negative predictive value of over 0.99 for prediction of microsatellite instability (MSI) on surgical resection specimens. We demonstrate that resection specimen-only training reaches clinical-grade performance on endoscopic biopsy tissue, solving a long-standing diagnostic problem.

Fibroglandular tissue segmentation in breast MRI using vision transformers: a multi-institutional evaluation

Accurate and automatic segmentation of fibroglandular tissue in breast MRI screening is essential for the quantification of breast density and background parenchymal enhancement. In this retrospective study, we developed and evaluated a transformer-based neural network for breast segmentation (TraBS) in multi-institutional MRI data, and compared its performance to the well established convolutional neural network nnUNet. TraBS and nnUNet were trained and tested on 200 internal and 40 external breast MRI examinations using manual segmentations generated by experienced human readers. Segmentation performance was assessed in terms of the Dice score and the average symmetric surface distance. The Dice score for nnUNet was lower than for TraBS on the internal testset (0.909 ± 0.069 versus 0.916 ± 0.067, P < 0.001) and on the external testset (0.824 ± 0.144 versus 0.864 ± 0.081, P = 0.004). Moreover, the average symmetric surface distance was higher (= worse) for nnUNet than for TraBS on the internal (0.657 ± 2.856 versus 0.548 ± 2.195, P = 0.001) and on the external testset (0.727 ± 0.620 versus 0.584 ± 0.413, P = 0.03). Our study demonstrates that transformer-based networks improve the quality of fibroglandular tissue segmentation in breast MRI compared to convolutional-based models like nnUNet. These findings might help to enhance the accuracy of breast density and parenchymal enhancement quantification in breast MRI screening.

Optical coherence tomography combined with convolutional neural networks can differentiate between intrahepatic cholangiocarcinoma and liver parenchyma ex vivo

PURPOSE

Surgical resection with complete tumor excision (R0) provides the best chance of long-term survival for patients with intrahepatic cholangiocarcinoma (iCCA). A non-invasive imaging technology, which could provide quick intraoperative assessment of resection margins, as an adjunct to histological examination, is optical coherence tomography (OCT). In this study, we investigated the ability of OCT combined with convolutional neural networks (CNN), to differentiate iCCA from normal liver parenchyma ex vivo.

METHODS

Consecutive adult patients undergoing elective liver resections for iCCA between June 2020 and April 2021 (n = 11) were included in this study. Areas of interest from resection specimens were scanned ex vivo, before formalin fixation, using a table-top OCT device at 1310 nm wavelength. Scanned areas were marked and histologically examined, providing a diagnosis for each scan. An Xception CNN was trained, validated, and tested in matching OCT scans to their corresponding histological diagnoses, through a 5 × 5 stratified cross-validation process.

RESULTS

Twenty-four three-dimensional scans (corresponding to approx. 85,603 individual) from ten patients were included in the analysis. In 5 × 5 cross-validation, the model achieved a mean F1-score, sensitivity, and specificity of 0.94, 0.94, and 0.93, respectively.

CONCLUSION

Optical coherence tomography combined with CNN can differentiate iCCA from liver parenchyma ex vivo. Further studies are necessary to expand on these results and lead to innovative in vivo OCT applications, such as intraoperative or endoscopic scanning.

A multimodal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical image synthesis

Although generative adversarial networks (GANs) can produce large datasets, their limited diversity and fidelity have been recently addressed by denoising diffusion probabilistic models, which have demonstrated superiority in natural image synthesis. In this study, we introduce Medfusion, a conditional latent DDPM designed for medical image generation, and evaluate its performance against GANs, which currently represent the state-of-the-art. Medfusion was trained and compared with StyleGAN-3 using fundoscopy images from the AIROGS dataset, radiographs from the CheXpert dataset, and histopathology images from the CRCDX dataset. Based on previous studies, Progressively Growing GAN (ProGAN) and Conditional GAN (cGAN) were used as additional baselines on the CheXpert and CRCDX datasets, respectively. Medfusion exceeded GANs in terms of diversity (recall), achieving better scores of 0.40 compared to 0.19 in the AIROGS dataset, 0.41 compared to 0.02 (cGAN) and 0.24 (StyleGAN-3) in the CRMDX dataset, and 0.32 compared to 0.17 (ProGAN) and 0.08 (StyleGAN-3) in the CheXpert dataset. Furthermore, Medfusion exhibited equal or higher fidelity (precision) across all three datasets. Our study shows that Medfusion constitutes a promising alternative to GAN-based models for generating high-quality medical images, leading to improved diversity and less artifacts in the generated images.

Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data

When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available.

An MRI Deep Learning Model Predicts Outcome in Rectal Cancer

Background Deep learning (DL) models can potentially improve prognostication of rectal cancer but have not been systematically assessed. Purpose To develop and validate an MRI DL model for predicting survival in patients with rectal cancer based on segmented tumor volumes from pretreatment T2-weighted MRI scans. Materials and Methods DL models were trained and validated on retrospectively collected MRI scans of patients with rectal cancer diagnosed between August 2003 and April 2021 at two centers. Patients were excluded from the study if there were concurrent malignant neoplasms, prior anticancer treatment, incomplete course of neoadjuvant therapy, or no radical surgery performed. The Harrell C-index was used to determine the best model, which was applied to internal and external test sets. Patients were stratified into high- and low-risk groups based on a fixed cutoff calculated in the training set. A multimodal model was also assessed, which used DL model-computed risk score and pretreatment carcinoembryonic antigen level as input. Results The training set included 507 patients (median age, 56 years [IQR, 46-64 years]; 355 men). In the validation set (n = 218; median age, 55 years [IQR, 47-63 years]; 144 men), the best algorithm reached a C-index of 0.82 for overall survival. The best model reached hazard ratios of 3.0 (95% CI: 1.0, 9.0) in the high-risk group in the internal test set (n = 112; median age, 60 years [IQR, 52-70 years]; 76 men) and 2.3 (95% CI: 1.0, 5.4) in the external test set (n = 58; median age, 57 years [IQR, 50-67 years]; 38 men). The multimodal model further improved the performance, with a C-index of 0.86 and 0.67 for the validation and external test set, respectively. Conclusion A DL model based on preoperative MRI was able to predict survival of patients with rectal cancer. The model could be used as a preoperative risk stratification tool. Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Langs in this issue.

Dual center validation of deep learning for automated multi-label segmentation of thoracic anatomy in bedside chest radiographs

BACKGROUND AND OBJECTIVES

Bedside chest radiographs (CXRs) are challenging to interpret but important for monitoring cardiothoracic disease and invasive therapy devices in critical care and emergency medicine. Taking surrounding anatomy into account is likely to improve the diagnostic accuracy of artificial intelligence and bring its performance closer to that of a radiologist. Therefore, we aimed to develop a deep convolutional neural network for efficient automatic anatomy segmentation of bedside CXRs.

METHODS

To improve the efficiency of the segmentation process, we introduced a "human-in-the-loop" segmentation workflow with an active learning approach, looking at five major anatomical structures in the chest (heart, lungs, mediastinum, trachea, and clavicles). This allowed us to decrease the time needed for segmentation by 32% and select the most complex cases to utilize human expert annotators efficiently. After annotation of 2,000 CXRs from different Level 1 medical centers at Charité - University Hospital Berlin, there was no relevant improvement in model performance, and the annotation process was stopped. A 5-layer U-ResNet was trained for 150 epochs using a combined soft Dice similarity coefficient (DSC) and cross-entropy as a loss function. DSC, Jaccard index (JI), Hausdorff distance (HD) in mm, and average symmetric surface distance (ASSD) in mm were used to assess model performance. External validation was performed using an independent external test dataset from Aachen University Hospital (n = 20).

RESULTS

The final training, validation, and testing dataset consisted of 1900/50/50 segmentation masks for each anatomical structure. Our model achieved a mean DSC/JI/HD/ASSD of 0.93/0.88/32.1/5.8 for the lung, 0.92/0.86/21.65/4.85 for the mediastinum, 0.91/0.84/11.83/1.35 for the clavicles, 0.9/0.85/9.6/2.19 for the trachea, and 0.88/0.8/31.74/8.73 for the heart. Validation using the external dataset showed an overall robust performance of our algorithm.

CONCLUSIONS

Using an efficient computer-aided segmentation method with active learning, our anatomy-based model achieves comparable performance to state-of-the-art approaches. Instead of only segmenting the non-overlapping portions of the organs, as previous studies did, a closer approximation to actual anatomy is achieved by segmenting along the natural anatomical borders. This novel anatomy approach could be useful for developing pathology models for accurate and quantifiable diagnosis.

Denoising diffusion probabilistic models for 3D medical image generation

Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen, and Stable Diffusion. However, their use in medicine, where imaging data typically comprises three-dimensional volumes, has not been systematically evaluated. Synthetic images may play a crucial role in privacy-preserving artificial intelligence and can also be used to augment small datasets. We show that diffusion probabilistic models can synthesize high-quality medical data for magnetic resonance imaging (MRI) and computed tomography (CT). For quantitative evaluation, two radiologists rated the quality of the synthesized images regarding "realistic image appearance", "anatomical correctness", and "consistency between slices". Furthermore, we demonstrate that synthetic images can be used in self-supervised pre-training and improve the performance of breast segmentation models when data is scarce (Dice scores, 0.91 [without synthetic data], 0.95 [with synthetic data]).

Collaborative training of medical artificial intelligence models with non-uniform labels

Due to the rapid advancements in recent years, medical image analysis is largely dominated by deep learning (DL). However, building powerful and robust DL models requires training with large multi-party datasets. While multiple stakeholders have provided publicly available datasets, the ways in which these data are labeled vary widely. For Instance, an institution might provide a dataset of chest radiographs containing labels denoting the presence of pneumonia, while another institution might have a focus on determining the presence of metastases in the lung. Training a single AI model utilizing all these data is not feasible with conventional federated learning (FL). This prompts us to propose an extension to the widespread FL process, namely flexible federated learning (FFL) for collaborative training on such data. Using 695,000 chest radiographs from five institutions from across the globe-each with differing labels-we demonstrate that having heterogeneously labeled datasets, FFL-based training leads to significant performance increase compared to conventional FL training, where only the uniformly annotated images are utilized. We believe that our proposed algorithm could accelerate the process of bringing collaborative training methods from research and simulation phase to the real-world applications in healthcare.

The Impact of Body Mass Composition on Outcome in Multiple Traumatized Patients—Results from the Fourth Thoracic and Third Lumbar Vertebrae: A Single-Center Retrospective Observational Study

Background: Body mass composition (BC) was shown to correlate with outcome in patients after surgery and minor trauma. As BC is assessed using computed tomography (CT) and routinely applied in multiple trauma (MT), this study will help to analyze whether BC variables also correlate with outcome in trauma patients. Materials and Methods: Inclusion criteria were MT (Injury Severity Score (ISS) > 15) and whole-body CT (WBCT) scan on admission. Muscle and fat tissue were assessed at the level of the fourth thoracic vertebra (T4) and the third lumbar vertebra (L3) using Slice-O-matic software, version 5.0 (Tomovision, Montreal, QC, Canada). Univariate and multivariate regression models were used with regard to outcome parameters such as duration of ventilation, hospital stay, local (i.e., pneumonia, wound infection) and systemic (i.e., MODS, SIRS) complications, and mortality. Results: 297 patients were included. BC correlated with both the development and severity of complications. Skeletal muscle index (SMI) and subcutaneous adipose tissue index (SATI) at both T4 and L3 correlated positively with the occurrence of systemic infections. Local infections positively correlated with SMI at T4. Low muscle mass and high visceral adipose tissue (VAT) predicted the severity of systemic and local complications. Muscle tissue markers at both T4 and L3 predicted the severity of complications in roughly the same way. Moreover, higher muscle mass at the L3 level was significantly associated with higher overall survival, while SATI at the T4 level correlated positively with hospital stay, length of stay in the ICU, and duration of ventilation. Conclusions: A lower muscle mass and a high adipose tissue index are associated with a poor outcome in MT. For the first time, it was shown that BC at the fourth thoracic vertebra is associated with comparable results to those found at the third lumbar level.

Using Machine Learning to Reduce the Need for Contrast Agents in Breast MRI through Synthetic Images

Background Reducing the amount of contrast agent needed for contrast-enhanced breast MRI is desirable. Purpose To investigate if generative adversarial networks (GANs) can recover contrast-enhanced breast MRI scans from unenhanced images and virtual low-contrast-enhanced images. Materials and Methods In this retrospective study of breast MRI performed from January 2010 to December 2019, simulated low-contrast images were produced by adding virtual noise to the existing contrast-enhanced images. GANs were then trained to recover the contrast-enhanced images from the simulated low-contrast images (approach A) or from the unenhanced T1- and T2-weighted images (approach B). Two experienced radiologists were tasked with distinguishing between real and synthesized contrast-enhanced images using both approaches. Image appearance and conspicuity of enhancing lesions on the real versus synthesized contrast-enhanced images were independently compared and rated on a five-point Likert scale. P values were calculated by using bootstrapping. Results A total of 9751 breast MRI examinations from 5086 patients (mean age, 56 years ± 10 [SD]) were included. Readers who were blinded to the nature of the images could not distinguish real from synthetic contrast-enhanced images (average accuracy of differentiation: approach A, 52 of 100; approach B, 61 of 100). The test set included images with and without enhancing lesions (29 enhancing masses and 21 nonmass enhancement; 50 total). When readers who were not blinded compared the appearance of the real versus synthetic contrast-enhanced images side by side, approach A image ratings were significantly higher than those of approach B (mean rating, 4.6 ± 0.1 vs 3.0 ± 0.2; P < .001), with the noninferiority margin met by synthetic images from approach A (P < .001) but not B (P > .99). Conclusion Generative adversarial networks may be useful to enable breast MRI with reduced contrast agent dose. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Bahl in this issue.

What Does DALL-E 2 Know About Radiology?

Generative models, such as DALL-E 2 (OpenAI), could represent promising future tools for image generation, augmentation, and manipulation for artificial intelligence research in radiology, provided that these models have sufficient medical domain knowledge. Herein, we show that DALL-E 2 has learned relevant representations of x-ray images, with promising capabilities in terms of zero-shot text-to-image generation of new images, the continuation of an image beyond its original boundaries, and the removal of elements; however, its capabilities for the generation of images with pathological abnormalities (eg, tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited. The use of generative models for augmenting and generating radiological data thus seems feasible, even if the further fine-tuning and adaptation of these models to their respective domains are required first.

Generalizable biomarker prediction from cancer pathology slides with self-supervised deep learning: A retrospective multi-centric study

Deep learning (DL) can predict microsatellite instability (MSI) from routine histopathology slides of colorectal cancer (CRC). However, it is unclear whether DL can also predict other biomarkers with high performance and whether DL predictions generalize to external patient populations. Here, we acquire CRC tissue samples from two large multi-centric studies. We systematically compare six different state-of-the-art DL architectures to predict biomarkers from pathology slides, including MSI and mutations in BRAF, KRAS, NRAS, and PIK3CA. Using a large external validation cohort to provide a realistic evaluation setting, we show that models using self-supervised, attention-based multiple-instance learning consistently outperform previous approaches while offering explainable visualizations of the indicative regions and morphologies. While the prediction of MSI and BRAF mutations reaches a clinical-grade performance, mutation prediction of PIK3CA, KRAS, and NRAS was clinically insufficient.

Getting Cartilage Thickness Measurements Right: A Systematic Inter-Method Comparison Using MRI Data from the Osteoarthritis Initiative

OBJECTIVE

Magnetic resonance imaging is the standard imaging modality to assess articular cartilage. As the imaging surrogate of degenerative joint disease, cartilage thickness is commonly quantified after tissue segmentation. In lack of a standard method, this study systematically compared five methods for automatic cartilage thickness measurements across the knee joint and as a function of region and sub-region: 3D mesh normals (3D-MN), 3D nearest neighbors (3D-NN), 3D ray tracing (3D-RT), 2D centerline normals (2D-CN), and 2D surface normals (2D-SN).

DESIGN

Based on the manually segmented femoral and tibial cartilage of 507 human knee joints, mean cartilage thickness was computed for the entire femorotibial joint, 4 joint regions, and 20 subregions using these methods. Inter-method comparisons of mean cartilage thickness and computation times were performed by one-way analysis of variance (ANOVA), Bland-Altman analyses and Lin's concordance correlation coefficient (CCC).

RESULTS

Mean inter-method differences in cartilage thickness were significant in nearly all subregions (P < 0.001). By trend, mean differences were smallest between 3D-MN and 2D-SN in most (sub)regions, which is also reflected by highest quantitative inter-method agreement and CCCs. 3D-RT was prone to severe overestimation of up to 2.5 mm. 3D-MN, 3D-NN, and 2D-SN required mean processing times of ≤5.3 s per joint and were thus similarly efficient, whereas the time demand of 2D-CN and 3D-RT was much larger at 133 ± 29 and 351 ± 10 s per joint (P < 0.001).

CONCLUSIONS

In automatic cartilage thickness determination, quantification accuracy and computational burden are largely affected by the underlying method. Mesh and surface normals or nearest neighbor searches should be used because they accurately capture variable geometries while being time-efficient.

Artificial Intelligence for Clinical Interpretation of Bedside Chest Radiographs

Background Supine chest radiography for bedridden patients in intensive care units (ICUs) is one of the most frequently ordered imaging studies worldwide. Purpose To evaluate the diagnostic performance of a neural network-based model that is trained on structured semiquantitative radiologic reports of bedside chest radiographs. Materials and Methods For this retrospective single-center study, children and adults in the ICU of a university hospital who had been imaged using bedside chest radiography from January 2009 to December 2020 were reported by using a structured and itemized template. Ninety-eight radiologists rated the radiographs semiquantitatively for the severity of disease patterns. These data were used to train a neural network to identify cardiomegaly, pulmonary congestion, pleural effusion, pulmonary opacities, and atelectasis. A held-out internal test set (100 radiographs from 100 patients) that was assessed independently by an expert panel of six radiologists provided the ground truth. Individual assessments by each of these six radiologists, by two nonradiologist physicians in the ICU, and by the neural network were compared with the ground truth. Separately, the nonradiologist physicians assessed the images without and with preliminary readings provided by the neural network. The weighted Cohen κ coefficient was used to measure agreement between the readers and the ground truth. Results A total of 193 566 radiographs in 45 016 patients (mean age, 66 years ± 16 [SD]; 61% men) were included and divided into training (n = 122 294; 64%), validation (n = 31 243; 16%), and test (n = 40 029; 20%) sets. The neural network exhibited higher agreement with a majority vote of the expert panel (κ = 0.86) than each individual radiologist compared with the majority vote of the expert panel (κ = 0.81 to ≤0.84). When the neural network provided preliminary readings, the reports of the nonradiologist physicians improved considerably (aided vs unaided, κ = 0.87 vs 0.79, respectively; P < .001). Conclusion A neural network trained with structured semiquantitative bedside chest radiography reports allowed nonradiologist physicians improved interpretations compared with the consensus reading of expert radiologists. © RSNA, 2022 Supplemental material is available for this article. See also the editorial by Wielpütz in this issue.

Adversarial attacks and adversarial robustness in computational pathology

Artificial Intelligence (AI) can support diagnostic workflows in oncology by aiding diagnosis and providing biomarkers directly from routine pathology slides. However, AI applications are vulnerable to adversarial attacks. Hence, it is essential to quantify and mitigate this risk before widespread clinical use. Here, we show that convolutional neural networks (CNNs) are highly susceptible to white- and black-box adversarial attacks in clinically relevant weakly-supervised classification tasks. Adversarially robust training and dual batch normalization (DBN) are possible mitigation strategies but require precise knowledge of the type of attack used in the inference. We demonstrate that vision transformers (ViTs) perform equally well compared to CNNs at baseline, but are orders of magnitude more robust to white- and black-box attacks. At a mechanistic level, we show that this is associated with a more robust latent representation of clinically relevant categories in ViTs compared to CNNs. Our results are in line with previous theoretical studies and provide empirical evidence that ViTs are robust learners in computational pathology. This implies that large-scale rollout of AI models in computational pathology should rely on ViTs rather than CNN-based classifiers to provide inherent protection against perturbation of the input data, especially adversarial attacks.

Artificial Intelligence can support diagnostic workflows in oncology, but they are vulnerable to adversarial attacks. Here, the authors show that convolutional neural networks are highly susceptible to white- and black-box adversarial attacks in clinically relevant classification tasks.

Possible Alterations of Imaging Patterns in Computed Tomography for Delta-VOC of SARS-CoV-2

BACKGROUND

 So far, typical findings for COVID-19 in computed tomography (CT) have been described as bilateral, multifocal ground glass opacities (GGOs) and consolidations, as well as intralobular and interlobular septal thickening. On the contrary, round consolidations with the halo sign are considered uncommon and are typically found in fungal infections, such as invasive pulmonary aspergillosis. The authors recently observed several patients with COVID-19 pneumonia presenting with round, multifocal consolidations accompanied by a halo sign. As this may indicate alterations of CT morphology based on the virus variant, the aim of this study was to investigate this matter in more detail.

METHODS

 161 CT scans of patients with confirmed SARS-CoV-2 infection (RT-PCR within 2 days of CT) examined between January 2021 and September 15, 2021 were included. Follow-up examinations, patients with invasive ventilation at the time of CT, and patients with insufficient virus typing for variants of concern (VOC) were excluded. CT scans were assessed for vertical and axial distribution of pulmonary patterns, degree of involvement, uni- vs. bilaterality, reticulations, and other common findings. The mean density of representative lesions was assessed in Hounsfield units. Results were compared using Mann-Whitney U-tests, Student's t-rests, descriptive statistics, and Fisher's exact tests.

RESULTS

 75 patients did not meet the inclusion criteria. Therefore, 86/161 CT scans of unique patients were analyzed. PCR VOC testing confirmed manifestation of the Delta-VOC SARS-CoV-2 in 22 patients, 39 patients with Alpha-VOC and the remaining 25 patients with Non-VOC SARS-CoV-2 infections. Three patients with the Delta-VOC demonstrated multiple pulmonary masses or nodules with surrounding halo sign, whereas no patients with either Alpha-VOC (p = 0.043) or non-VOC (p = 0.095) demonstrated these findings. All three patients were admitted to normal wards and had no suspicion of a pulmonary co-infection. Patients with Delta-VOC were less likely to have ground glass opacities compared to Alpha-VOC (7/22 or 31.8 % vs. 4/39 or 10.3 %; p < 0.001), whereas a significant difference has not been observed between Delta-VOC and non-VOC (5/25 or 20 %; p = 0.348). The mean representative density of lesions did not show significant differences between the studied cohorts.

CONCLUSION

 In this study 3 out of 22 patients (13.6 %) with Delta-VOC presented with bilateral round pulmonary masses or nodules with surrounding halo signs, which has not been established as a notable imaging pattern in COVID-19 pneumonia yet. Compared to the other cohorts, a lesser percentage of patients with Delta-VOC presented with ground glass opacities. Based on these results Delta-VOC might cause a divergence in CT-morphologic phenotype.

KEY POINTS

· Until recently, CT-morphologic signs of COVID-19 pneumonia have been presumed to be uncontroversially understood. Yet, recently the authors observed diverging pulmonary alterations in patients infected with Delta-VOC.. · These imaging alterations included round pulmonary masses or nodules with surrounding halo sign.. · These imaging alterations have not yet been established as typical for COVID-19 pneumonia, yet.. · Based on these results, Delta-VOC could impose a divergence of CT-morphologic phenotype..

CITATION FORMAT

· Yüksel C, Sähn M, Kleines M et al. Possible Alterations of Imaging Patterns in Computed Tomography for Delta-VOC of SARS-CoV-2 . Fortschr Röntgenstr 2022; DOI: 10.1055/a-1826-0436.