Deep learning for diagnosis and survival prediction in soft tissue sarcoma

Utility of artificial intelligence in a binary classification of soft tissue tumors

Soft tissue tumors (STTs) pose diagnostic and therapeutic challenges due to their rarity, complexity, and morphological overlap. Accurate differentiation between benign and malignant STTs is important to set treatment directions, however, this task can be difficult. The integration of machine learning and artificial intelligence (AI) models can potentially be helpful in classifying these tumors. The aim of this study was to investigate AI and machine learning tools in the classification of STT into benign and malignant categories. This study consisted of three components: (1) Evaluation of whole-slide images (WSIs) to classify STT into benign and malignant entities. Five specialized soft tissue pathologists from different medical centers independently reviewed 100 WSIs, representing 100 different cases, with limited clinical information and no additional workup. The results showed an overall concordance rate of 70.4% compared to the reference diagnosis. (2) Identification of cell-specific parameters that can distinguish benign and malignant STT. Using an image analysis software (QuPath) and a cohort of 95 cases, several cell-specific parameters were found to be statistically significant, most notably cell count, nucleus/cell area ratio, nucleus hematoxylin density mean, and cell max caliper. (3) Evaluation of machine learning library (Scikit-learn) in differentiating benign and malignant STTs. A total of 195 STT cases (156 cases in the training group and 39 cases in the validation group) achieved approximately 70% sensitivity and specificity, and an AUC of 0.68. Our limited study suggests that the use of WSI and AI in soft tissue pathology has the potential to enhance diagnostic accuracy and identify parameters that can differentiate between benign and malignant STTs. We envision the integration of AI as a supportive tool to augment the pathologists' diagnostic capabilities.

Prediction of non-muscle invasive bladder cancer recurrence using deep learning of pathology image

We aimed to build a deep learning-based pathomics model to predict the early recurrence of non-muscle-infiltrating bladder cancer (NMIBC) in this work. A total of 147 patients from Xuzhou Central Hospital were enrolled as the training cohort, and 63 patients from Suqian Affiliated Hospital of Xuzhou Medical University were enrolled as the test cohort. Based on two consecutive phases of patch level prediction and WSI-level predictione, we built a pathomics model, with the initial model developed in the training cohort and subjected to transfer learning, and then the test cohort was validated for generalization. The features extracted from the visualization model were used for model interpretation. After migration learning, the area under the receiver operating characteristic curve for the deep learning-based pathomics model in the test cohort was 0.860 (95% CI 0.752-0.969), with good agreement between the migration training cohort and the test cohort in predicting recurrence, and the predicted values matched well with the observed values, with p values of 0.667766 and 0.140233 for the Hosmer-Lemeshow test, respectively. The good clinical application was observed using a decision curve analysis method. We developed a deep learning-based pathomics model showed promising performance in predicting recurrence within one year in NMIBC patients. Including 10 state prediction NMIBC recurrence group pathology features be visualized, which may be used to facilitate personalized management of NMIBC patients to avoid ineffective or unnecessary treatment for the benefit of patients.

Deep learning predicts the 1-year prognosis of pancreatic cancer patients using positive peritoneal washing cytology

Peritoneal washing cytology (CY) in patients with pancreatic cancer is mainly used for staging; however, it may also be used to evaluate the intraperitoneal status to predict a more accurate prognosis. Here, we investigated the potential of deep learning of CY specimen images for predicting the 1-year prognosis of pancreatic cancer in CY-positive patients. CY specimens from 88 patients with prognostic information were retrospectively analyzed. CY specimens scanned by the whole slide imaging device were segmented and subjected to deep learning with a Vision Transformer (ViT) and a Convolutional Neural Network (CNN). The results indicated that ViT and CNN predicted the 1-year prognosis from scanned images with accuracies of 0.8056 and 0.8009 in the area under the curve of the receiver operating characteristic curves, respectively. Patients predicted to survive 1 year or more by ViT showed significantly longer survivals by Kaplan-Meier analyses. The cell nuclei found to have a negative prognostic impact by ViT appeared to be neutrophils. Our results indicate that AI-mediated analysis of CY specimens can successfully predict the 1-year prognosis of patients with pancreatic cancer positive for CY. Intraperitoneal neutrophils may be a novel prognostic marker and therapeutic target for CY-positive patients with pancreatic cancer.

A multi-classifier system integrated by clinico-histology-genomic analysis for predicting recurrence of papillary renal cell carcinoma

Integrating genomics and histology for cancer prognosis demonstrates promise. Here, we develop a multi-classifier system integrating a lncRNA-based classifier, a deep learning whole-slide-image-based classifier, and a clinicopathological classifier to accurately predict post-surgery localized (stage I-III) papillary renal cell carcinoma (pRCC) recurrence. The multi-classifier system demonstrates significantly higher predictive accuracy for recurrence-free survival (RFS) compared to the three single classifiers alone in the training set and in both validation sets (C-index 0.831-0.858 vs. 0.642-0.777, p < 0.05). The RFS in our multi-classifier-defined high-risk stage I/II and grade 1/2 groups is significantly worse than in the low-risk stage III and grade 3/4 groups (p < 0.05). Our multi-classifier system is a practical and reliable predictor for recurrence of localized pRCC after surgery that can be used with the current staging system to more accurately predict disease course and inform strategies for individualized adjuvant therapy.

Prediction of HER2 Status Based on Deep Learning in H&E-Stained Histopathology Images of Bladder Cancer

Epidermal growth factor receptor 2 (HER2) has been widely recognized as one of the targets for bladder cancer immunotherapy. The key to implementing personalized treatment for bladder cancer patients lies in achieving rapid and accurate diagnosis. To tackle this challenge, we have pioneered the application of deep learning techniques to predict HER2 expression status from H&E-stained pathological images of bladder cancer, bypassing the need for intricate IHC staining or high-throughput sequencing methods. Our model, when subjected to rigorous testing within the cohort from the People's Hospital of Wuhan University, which encompasses 106 cases, has exhibited commendable performance on both the validation and test datasets. Specifically, the validation set yielded an AUC of 0.92, an accuracy of 0.86, a sensitivity of 0.87, a specificity of 0.83, and an F1 score of 86.7%. The corresponding metrics for the test set were 0.88 for AUC, 0.67 for accuracy, 0.56 for sensitivity, 0.75 for specificity, and 77.8% for F1 score. Additionally, in a direct comparison with pathologists, our model demonstrated statistically superior performance, with a p-value less than 0.05, highlighting its potential as a powerful diagnostic tool.

Machine Learning-Assisted Decision Making in Orthopaedic Oncology

» Artificial intelligence is an umbrella term for computational calculations that are designed to mimic human intelligence and problem-solving capabilities, although in the future, this may become an incomplete definition. Machine learning (ML) encompasses the development of algorithms or predictive models that generate outputs without explicit instructions, assisting in clinical predictions based on large data sets. Deep learning is a subset of ML that utilizes layers of networks that use various inter-relational connections to define and generalize data.» ML algorithms can enhance radiomics techniques for improved image evaluation and diagnosis. While ML shows promise with the advent of radiomics, there are still obstacles to overcome.» Several calculators leveraging ML algorithms have been developed to predict survival in primary sarcomas and metastatic bone disease utilizing patient-specific data. While these models often report exceptionally accurate performance, it is crucial to evaluate their robustness using standardized guidelines.» While increased computing power suggests continuous improvement of ML algorithms, these advancements must be balanced against challenges such as diversifying data, addressing ethical concerns, and enhancing model interpretability.

EfficientNet-Based System for Detecting EGFR-Mutant Status and Predicting Prognosis of Tyrosine Kinase Inhibitors in Patients with NSCLC

We aimed to develop and validate a deep learning-based system using pre-therapy computed tomography (CT) images to detect epidermal growth factor receptor (EGFR)-mutant status in patients with non-small cell lung cancer (NSCLC) and predict the prognosis of advanced-stage patients with EGFR mutations treated with EGFR tyrosine kinase inhibitors (TKI). This retrospective, multicenter study included 485 patients with NSCLC from four hospitals. Of them, 339 patients from three centers were included in the training dataset to develop an EfficientNetV2-L-based model (EME) for predicting EGFR-mutant status, and the remaining patients were assigned to an independent test dataset. EME semantic features were extracted to construct an EME-prognostic model to stratify the prognosis of EGFR-mutant NSCLC patients receiving EGFR-TKI. A comparison of EME and radiomics was conducted. Additionally, we included patients from The Cancer Genome Atlas lung adenocarcinoma dataset with both CT images and RNA sequencing data to explore the biological associations between EME score and EGFR-related biological processes. EME obtained an area under the curve (AUC) of 0.907 (95% CI 0.840-0.926) on the test dataset, superior to the radiomics model (P = 0.007). The EME and radiomics fusion model showed better (AUC, 0.941) but not significantly increased performance (P = 0.895) compared with EME. In prognostic stratification, the EME-prognostic model achieved the best performance (C-index, 0.711). Moreover, the EME-prognostic score showed strong associations with biological pathways related to EGFR expression and EGFR-TKI efficacy. EME demonstrated a non-invasive and biologically interpretable approach to predict EGFR status, stratify survival prognosis, and correlate biological pathways in patients with NSCLC.

Predicting overall survival and prophylactic cranial irradiation benefit in small-cell lung cancer with CT-based deep learning: A retrospective multicenter study

BACKGROUND AND PURPOSE

To develop a computed tomography (CT)-based deep learning model to predict overall survival (OS) among small-cell lung cancer (SCLC) patients and identify patients who could benefit from prophylactic cranial irradiation (PCI) based on OS signature risk stratification.

MATERIALS AND METHODS

This study retrospectively included 556 SCLC patients from three medical centers. The training, internal validation, and external validation cohorts comprised 309, 133, and 114 patients, respectively. The OS signature was built using a unified fully connected neural network. A deep learning model was developed based on the OS signature. Clinical and combined models were developed and compared with a deep learning model. Additionally, the benefits of PCI were evaluated after stratification using an OS signature.

RESULTS

Within the internal and external validation cohorts, the deep learning model (concordance index [C-index] 0.745, 0.733) was far superior to the clinical model (C-index: 0.635, 0.630) in predicting OS, but slightly worse than the combined model (C-index: 0.771, 0.770). Additionally, the deep learning model had excellent calibration, clinical usefulness, and improved accuracy in classifying survival outcomes. Remarkably, patients at high risk had a survival benefit from PCI in both the limited and extensive stages (all P < 0.05), whereas no significant association was observed in patients at low risk.

CONCLUSIONS

The CT-based deep learning model exhibited promising performance in predicting the OS of SCLC patients. The OS signature may aid in individualized treatment planning to select patients who may benefit from PCI.

Prediction of prognosis using artificial intelligence-based histopathological image analysis in patients with soft tissue sarcomas

BACKGROUND

Prompt histopathological diagnosis with accuracy is required for soft tissue sarcomas (STSs) which are still challenging. In addition, the advances in artificial intelligence (AI) along with the development of pathology slides digitization may empower the demand for the prediction of behavior of STSs. In this article, we explored the application of deep learning for prediction of prognosis from histopathological images in patients with STS.

METHODS

Our retrospective study included a total of 35 histopathological slides from patients with STS. We trained Inception v3 which is proposed method of convolutional neural network based survivability estimation. F1 score which identify the accuracy and area under the receiver operating characteristic curve (AUC) served as main outcome measures from a 4-fold validation.

RESULTS

The cohort included 35 patients with a mean age of 64 years, and the mean follow-up period was 34 months (2-66 months). Our deep learning method achieved AUC of 0.974 and an accuracy of 91.9% in predicting overall survival. Concerning with the prediction of metastasis-free survival, the accuracy was 84.2% with the AUC of 0.852.

CONCLUSION

AI might be used to help pathologists with accurate prognosis prediction. This study could substantially improve the clinical management of patients with STS.

Deep learning framework for comprehensive molecular and prognostic stratifications of triple-negative breast cancer

Triple-negative breast cancer (TNBC) is the most challenging breast cancer subtype. Molecular stratification and target therapy bring clinical benefit for TNBC patients, but it is difficult to implement comprehensive molecular testing in clinical practice. Here, using our multi-omics TNBC cohort (N = 425), a deep learning-based framework was devised and validated for comprehensive predictions of molecular features, subtypes and prognosis from pathological whole slide images. The framework first incorporated a neural network to decompose the tissue on WSIs, followed by a second one which was trained based on certain tissue types for predicting different targets. Multi-omics molecular features were analyzed including somatic mutations, copy number alterations, germline mutations, biological pathway activities, metabolomics features and immunotherapy biomarkers. It was shown that the molecular features with therapeutic implications can be predicted including the somatic PIK3CA mutation, germline BRCA2 mutation and PD-L1 protein expression (area under the curve [AUC]: 0.78, 0.79 and 0.74 respectively). The molecular subtypes of TNBC can be identified (AUC: 0.84, 0.85, 0.93 and 0.73 for the basal-like immune-suppressed, immunomodulatory, luminal androgen receptor, and mesenchymal-like subtypes respectively) and their distinctive morphological patterns were revealed, which provided novel insights into the heterogeneity of TNBC. A neural network integrating image features and clinical covariates stratified patients into groups with different survival outcomes (log-rank P < 0.001). Our prediction framework and neural network models were externally validated on the TNBC cases from TCGA (N = 143) and appeared robust to the changes in patient population. For potential clinical translation, we built a novel online platform, where we modularized and deployed our framework along with the validated models. It can realize real-time one-stop prediction for new cases. In summary, using only pathological WSIs, our proposed framework can enable comprehensive stratifications of TNBC patients and provide valuable information for therapeutic decision-making. It had the potential to be clinically implemented and promote the personalized management of TNBC.

MOF@COF Nanocapsules Enhance Soft Tissue Sarcoma Treatment: Synergistic Effects of Photodynamic Therapy and PARP Inhibition on Tumor Growth Suppression and Immune Response Activation

Soft tissue sarcomas (STS) are highly malignant tumors with limited treatment options owing to their heterogeneity and resistance to conventional therapies. Photodynamic therapy (PDT) and poly-ADP-ribose polymerase (PARP) inhibitors (PARPi) have shown potential for STS treatment, with PDT being effective for sarcomas located on the extremities and body surface and PARPi targeting defects in homologous recombination repair. To address the limitations of PDT and harness the potential of PARPi, herein, a novel therapeutic approach for STS treatment combining nanocapsules bearing integrated metal-organic frameworks (MOFs) and covalent organic frameworks (COFs), i.e., MOF@COF, with PDT and PARPi is proposed. Nanocapsules are designed, referred to as ZTN@COF@poloxamer, which contain a Zr-based MOF and tetrakis (4-carbethoxyphenyl) porphyrin as a photosensitizer, are coated with a COF to improve the sensitizing properties, and are loaded with niraparib to inhibit DNA repair. Experiments demonstrate that this new nanocapsules treatment significantly inhibits STS growth, promotes tumor cell apoptosis, exhibits high antitumor activity with minimal side effects, activates the immune response of the tumor, and inhibits lung metastasis in vivo. Therefore, MOF@COF nanocapsules combined with PARPi offer a promising approach for STS treatment, with the potential to enhance the efficacy of PDT and prevent tumor recurrence.

Preliminary Evidence of the Use of Generative AI in Health Care Clinical Services: Systematic Narrative Review

BACKGROUND

Generative artificial intelligence tools and applications (GenAI) are being increasingly used in health care. Physicians, specialists, and other providers have started primarily using GenAI as an aid or tool to gather knowledge, provide information, train, or generate suggestive dialogue between physicians and patients or between physicians and patients' families or friends. However, unless the use of GenAI is oriented to be helpful in clinical service encounters that can improve the accuracy of diagnosis, treatment, and patient outcomes, the expected potential will not be achieved. As adoption continues, it is essential to validate the effectiveness of the infusion of GenAI as an intelligent technology in service encounters to understand the gap in actual clinical service use of GenAI.

OBJECTIVE

This study synthesizes preliminary evidence on how GenAI assists, guides, and automates clinical service rendering and encounters in health care The review scope was limited to articles published in peer-reviewed medical journals.

METHODS

We screened and selected 0.38% (161/42,459) of articles published between January 1, 2020, and May 31, 2023, identified from PubMed. We followed the protocols outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to select highly relevant studies with at least 1 element on clinical use, evaluation, and validation to provide evidence of GenAI use in clinical services. The articles were classified based on their relevance to clinical service functions or activities using the descriptive and analytical information presented in the articles.

RESULTS

Of 161 articles, 141 (87.6%) reported using GenAI to assist services through knowledge access, collation, and filtering. GenAI was used for disease detection (19/161, 11.8%), diagnosis (14/161, 8.7%), and screening processes (12/161, 7.5%) in the areas of radiology (17/161, 10.6%), cardiology (12/161, 7.5%), gastrointestinal medicine (4/161, 2.5%), and diabetes (6/161, 3.7%). The literature synthesis in this study suggests that GenAI is mainly used for diagnostic processes, improvement of diagnosis accuracy, and screening and diagnostic purposes using knowledge access. Although this solves the problem of knowledge access and may improve diagnostic accuracy, it is oriented toward higher value creation in health care.

CONCLUSIONS

GenAI informs rather than assisting or automating clinical service functions in health care. There is potential in clinical service, but it has yet to be actualized for GenAI. More clinical service-level evidence that GenAI is used to streamline some functions or provides more automated help than only information retrieval is needed. To transform health care as purported, more studies related to GenAI applications must automate and guide human-performed services and keep up with the optimism that forward-thinking health care organizations will take advantage of GenAI.

Research Progress of Artificial Intelligence in the Grading and Classification of Meningiomas

A meningioma is a common primary central nervous system tumor. The histological features of meningiomas vary significantly depending on the grade and subtype, leading to differences in treatment and prognosis. Therefore, early diagnosis, grading, and typing of meningiomas are crucial for developing comprehensive and individualized diagnosis and treatment plans. The advancement of artificial intelligence (AI) in medical imaging, particularly radiomics and deep learning (DL), has contributed to the increasing research on meningioma grading and classification. These techniques are fast and accurate, involve fully automated learning, are non-invasive and objective, enable the efficient and non-invasive prediction of meningioma grades and classifications, and provide valuable assistance in clinical treatment and prognosis. This article provides a summary and analysis of the research progress in radiomics and DL for meningioma grading and classification. It also highlights the existing research findings, limitations, and suggestions for future improvement, aiming to facilitate the future application of AI in the diagnosis and treatment of meningioma.

Emerging innovations and advancements in the treatment of extremity and truncal soft tissue sarcomas

In this special edition update on soft tissue sarcomas (STS), we cover classifications, emerging technologies, prognostic tools, radiation schemas, and treatment disparities in extremity and truncal STS. We discuss the importance of enhancing local control and reducing complications, including the role of innovative imaging, surgical guidance, and hypofractionated radiation. We review advancements in systemic and immunotherapeutic treatments and introduce disparities seen in this vulnerable population that must be considered to improve overall patient care.

[Multistain deep learning as a prognostic and predictive biomarker in colorectal cancer]

The tumor immune microenvironment (TIME) plays a crucial prognostic and predictive role in solid malignancies such as colorectal cancer (CRC). Nevertheless, scoring systems based on TIME such as the Immunoscore (IS) are rarely used in clinical practice. Among other reasons, this might be due to the additional time required for manual quantification of tumor-associated immune cells or costs associated with proprietary/commercial solutions. To address these issues, we developed a multistain deep learning model (MSDLM) and trained, validated, and tested it on immunohistochemical image data of different immune cell subtypes from over 1000 patients with CRC. Our model showed high prognostic accuracy and outperformed other clinical, molecular, and immune cell-based parameters. It might also be used for therapy response prediction in rectal cancer patients undergoing neoadjuvant therapy. Leveraging artificial intelligence interpretability/explainability methods, we ascertained that the MSDLM's predictions align with recognized antitumor immune response patterns. Consequently, the AImmunoscore (AIS) could emerge as a potential TIME-based decision-making tool for clinicians.

Standardized Classification of Lung Adenocarcinoma Subtypes and Improvement of Grading Assessment Through Deep Learning

The histopathologic distinction of lung adenocarcinoma (LADC) subtypes is subject to high interobserver variability, which can compromise the optimal assessment of patient prognosis. Therefore, this study developed convolutional neural networks capable of distinguishing LADC subtypes and predicting disease-specific survival, according to the recently established LADC tumor grades. Consensus LADC histopathologic images were obtained from 17 expert pulmonary pathologists and one pathologist in training. Two deep learning models (AI-1 and AI-2) were trained to predict eight different LADC classes. Furthermore, the trained models were tested on an independent cohort of 133 patients. The models achieved high precision, recall, and F1 scores exceeding 0.90 for most of the LADC classes. Clear stratification of the three LADC grades was reached in predicting the disease-specific survival by the two models, with both Kaplan-Meier curves showing significance (P = 0.0017 and 0.0003). Moreover, both trained models showed high stability in the segmentation of each pair of predicted grades with low variation in the hazard ratio across 200 bootstrapped samples. These findings indicate that the trained convolutional neural networks improve the diagnostic accuracy of the pathologist and refine LADC grade assessment. Thus, the trained models are promising tools that may assist in the routine evaluation of LADC subtypes and grades in clinical practice.

Determining the prognosis of Lung cancer from mutated genes using a deep learning survival model: a large multi-center study

BACKGROUND

Gene status has become the focus of prognosis prediction. Furthermore, deep learning has frequently been implemented in medical imaging to diagnose, prognosticate, and evaluate treatment responses in patients with cancer. However, few deep learning survival (DLS) models based on mutational genes that are directly associated with patient prognosis in terms of progression-free survival (PFS) or overall survival (OS) have been reported. Additionally, DLS models have not been applied to determine IO-related prognosis based on mutational genes. Herein, we developed a deep learning method to predict the prognosis of patients with lung cancer treated with or without immunotherapy (IO).

METHODS

Samples from 6542 patients from different centers were subjected to genome sequencing. A DLS model based on multi-panels of somatic mutations was trained and validated to predict OS in patients treated without IO and PFS in patients treated with IO.

RESULTS

In patients treated without IO, the DLS model (low vs. high DLS) was trained using the training MSK-MET cohort (HR = 0.241 [0.213-0.273], P < 0.001) and tested in the inter-validation MSK-MET cohort (HR = 0.175 [0.148-0.206], P < 0.001). The DLS model was then validated with the OncoSG, MSK-CSC, and TCGA-LUAD cohorts (HR = 0.420 [0.272-0.649], P < 0.001; HR = 0.550 [0.424-0.714], P < 0.001; HR = 0.215 [0.159-0.291], P < 0.001, respectively). Subsequently, it was fine-tuned and retrained in patients treated with IO. The DLS model (low vs. high DLS) could predict PFS and OS in the MIND, MSKCC, and POPLAR/OAK cohorts (P < 0.001, respectively). Compared with tumor-node-metastasis staging, the COX model, tumor mutational burden, and programmed death-ligand 1 expression, the DLS model had the highest C-index in patients treated with or without IO.

CONCLUSIONS

The DLS model based on mutational genes can robustly predict the prognosis of patients with lung cancer treated with or without IO.

Application possibilities of artificial intelligence in facial vascularized composite allotransplantation-a narrative review

Facial vascularized composite allotransplantation (FVCA) is an emerging field of reconstructive surgery that represents a dogmatic shift in the surgical treatment of patients with severe facial disfigurements. While conventional reconstructive strategies were previously considered the goldstandard for patients with devastating facial trauma, FVCA has demonstrated promising short- and long-term outcomes. Yet, there remain several obstacles that complicate the integration of FVCA procedures into the standard workflow for facial trauma patients. Artificial intelligence (AI) has been shown to provide targeted and resource-effective solutions for persisting clinical challenges in various specialties. However, there is a paucity of studies elucidating the combination of FVCA and AI to overcome such hurdles. Here, we delineate the application possibilities of AI in the field of FVCA and discuss the use of AI technology for FVCA outcome simulation, diagnosis and prediction of rejection episodes, and malignancy screening. This line of research may serve as a fundament for future studies linking these two revolutionary biotechnologies.

Novel metabolomics-biohumoral biomarkers model for predicting survival of metastatic soft-tissue sarcomas

Soft tissue sarcomas (STSs) are rare malignant tumors that are difficult to prognosticate using currently available instruments. Omics sciences could provide more accurate and individualized survival predictions for patients with metastatic STS. In this pilot, hypothesis-generating study, we integrated clinicopathological variables with proton nuclear magnetic resonance (1H NMR) plasma metabolomic and lipoproteomic profiles, capturing both tumor and host characteristics, to identify novel prognostic biomarkers of 2-year survival. Forty-five metastatic STS (mSTS) patients with prevalent leiomyosarcoma and liposarcoma histotypes receiving trabectedin treatment were enrolled. A score combining acetate, triglycerides low-density lipoprotein (LDL)-2, and red blood cell count was developed, and it predicts 2-year survival with optimal results in the present cohort (84.4% sensitivity, 84.6% specificity). This score is statistically significant and independent of other prognostic factors such as age, sex, tumor grading, tumor histotype, frailty status, and therapy administered. A nomogram based on these 3 biomarkers has been developed to inform the clinical use of the present findings.

Real-time detection of laryngopharyngeal cancer using an artificial intelligence-assisted system with multimodal data

BACKGROUND

Laryngopharyngeal cancer (LPC) includes laryngeal and hypopharyngeal cancer, whose early diagnosis can significantly improve the prognosis and quality of life of patients. Pathological biopsy of suspicious cancerous tissue under the guidance of laryngoscopy is the gold standard for diagnosing LPC. However, this subjective examination largely depends on the skills and experience of laryngologists, which increases the possibility of missed diagnoses and repeated unnecessary biopsies. We aimed to develop and validate a deep convolutional neural network-based Laryngopharyngeal Artificial Intelligence Diagnostic System (LPAIDS) for real-time automatically identifying LPC in both laryngoscopy white-light imaging (WLI) and narrow-band imaging (NBI) images to improve the diagnostic accuracy of LPC by reducing diagnostic variation among on-expert laryngologists.

METHODS

All 31,543 laryngoscopic images from 2382 patients were categorised into training, verification, and test sets to develop, validate, and internal test LPAIDS. Another 25,063 images from five other hospitals were used as external tests. Overall, 551 videos were used to evaluate the real-time performance of the system, and 200 randomly selected videos were used to compare the diagnostic performance of the LPAIDS with that of laryngologists. Two deep-learning models using either WLI (model W) or NBI (model N) images were constructed to compare with LPAIDS.

RESULTS

LPAIDS had a higher diagnostic performance than models W and N, with accuracies of 0·956 and 0·949 in the internal image and video tests, respectively. The robustness and stability of LPAIDS were validated in external sets with the area under the receiver operating characteristic curve values of 0·965-0·987. In the laryngologist-machine competition, LPAIDS achieved an accuracy of 0·940, which was comparable to expert laryngologists and outperformed other laryngologists with varying qualifications.

CONCLUSIONS

LPAIDS provided high accuracy and stability in detecting LPC in real-time, which showed great potential for using LPAIDS to improve the diagnostic accuracy of LPC by reducing diagnostic variation among on-expert laryngologists.

Development and Validation of a Deep Learning-Based Histologic Diagnosis System for Diagnosing Colorectal Sessile Serrated Lesions

OBJECTIVES

The histopathologic diagnosis of colorectal sessile serrated lesions (SSLs) and hyperplastic polyps (HPs) is of low consistency among pathologists. This study aimed to develop and validate a deep learning (DL)-based logical anthropomorphic pathology diagnostic system (LA-SSLD) for the differential diagnosis of colorectal SSL and HP.

METHODS

The diagnosis framework of the LA-SSLD system was constructed according to the current guidelines and consisted of 4 DL models. Deep convolutional neural network (DCNN) 1 was the mucosal layer segmentation model, DCNN 2 was the muscularis mucosa segmentation model, DCNN 3 was the glandular lumen segmentation model, and DCNN 4 was the glandular lumen classification (aberrant or regular) model. A total of 175 HP and 127 SSL sections were collected from Renmin Hospital of Wuhan University during November 2016 to November 2022. The performance of the LA-SSLD system was compared to 11 pathologists with different qualifications through the human-machine contest.

RESULTS

The Dice scores of DCNNs 1, 2, and 3 were 93.66%, 58.38%, and 74.04%, respectively. The accuracy of DCNN 4 was 92.72%. In the human-machine contest, the accuracy, sensitivity, and specificity of the LA-SSLD system were 85.71%, 86.36%, and 85.00%, respectively. In comparison with experts (pathologist D: accuracy 83.33%, sensitivity 90.91%, specificity 75.00%; pathologist E: accuracy 85.71%, sensitivity 90.91%, specificity 80.00%), LA-SSLD achieved expert-level accuracy and outperformed all the senior and junior pathologists.

CONCLUSIONS

This study proposed a logical anthropomorphic diagnostic system for the differential diagnosis of colorectal SSL and HP. The diagnostic performance of the system is comparable to that of experts and has the potential to become a powerful diagnostic tool for SSL in the future. It is worth mentioning that a logical anthropomorphic system can achieve expert-level accuracy with fewer samples, providing potential ideas for the development of other artificial intelligence models.

Predicting Non-Small-Cell Lung Cancer Survival after Curative Surgery via Deep Learning of Diffusion MRI

BACKGROUND

the objective of this study is to evaluate the predictive power of the survival model using deep learning of diffusion-weighted images (DWI) in patients with non-small-cell lung cancer (NSCLC).

METHODS

DWI at b-values of 0, 100, and 700 sec/mm2 (DWI0, DWI100, DWI700) were preoperatively obtained for 100 NSCLC patients who underwent curative surgery (57 men, 43 women; mean age, 62 years). The ADC0-100 (perfusion-sensitive ADC), ADC100-700 (perfusion-insensitive ADC), ADC0-100-700, and demographic features were collected as input data and 5-year survival was collected as output data. Our survival model adopted transfer learning from a pre-trained VGG-16 network, whereby the softmax layer was replaced with the binary classification layer for the prediction of 5-year survival. Three channels of input data were selected in combination out of DWIs and ADC images and their accuracies and AUCs were compared for the best performance during 10-fold cross validation.

RESULTS

66 patients survived, and 34 patients died. The predictive performance was the best in the following combination: DWI0-ADC0-100-ADC0-100-700 (accuracy: 92%; AUC: 0.904). This was followed by DWI0-DWI700-ADC0-100-700, DWI0-DWI100-DWI700, and DWI0-DWI0-DWI0 (accuracy: 91%, 81%, 76%; AUC: 0.889, 0.763, 0.711, respectively). Survival prediction models trained with ADC performed significantly better than the one trained with DWI only (p-values < 0.05). The survival prediction was improved when demographic features were added to the model with only DWIs, but the benefit of clinical information was not prominent when added to the best performing model using both DWI and ADC.

CONCLUSIONS

Deep learning may play a role in the survival prediction of lung cancer. The performance of learning can be enhanced by inputting precedented, proven functional parameters of the ADC instead of the original data of DWIs only.

Artificial Intelligence in the Pathology of Gastric Cancer

Recent advances in artificial intelligence (AI) have provided novel tools for rapid and precise pathologic diagnosis. The introduction of digital pathology has enabled the acquisition of scanned slide images that are essential for the application of AI. The application of AI for improved pathologic diagnosis includes the error-free detection of potentially negligible lesions, such as a minute focus of metastatic tumor cells in lymph nodes, the accurate diagnosis of potentially controversial histologic findings, such as very well-differentiated carcinomas mimicking normal epithelial tissues, and the pathological subtyping of the cancers. Additionally, the utilization of AI algorithms enables the precise decision of the score of immunohistochemical markers for targeted therapies, such as human epidermal growth factor receptor 2 and programmed death-ligand 1. Studies have revealed that AI assistance can reduce the discordance of interpretation between pathologists and more accurately predict clinical outcomes. Several approaches have been employed to develop novel biomarkers from histologic images using AI. Moreover, AI-assisted analysis of the cancer microenvironment showed that the distribution of tumor-infiltrating lymphocytes was related to the response to the immune checkpoint inhibitor therapy, emphasizing its value as a biomarker. As numerous studies have demonstrated the significance of AI-assisted interpretation and biomarker development, the AI-based approach will advance diagnostic pathology.

A Weakly Supervised Deep Learning Model and Human-Machine Fusion for Accurate Grading of Renal Cell Carcinoma from Histopathology Slides

(1) Background: The Fuhrman grading (FG) system is widely used in the management of clear cell renal cell carcinoma (ccRCC). However, it is affected by observer variability and irreproducibility in clinical practice. We aimed to use a deep learning multi-class model called SSL-CLAM to assist in diagnosing the FG status of ccRCC patients using digitized whole slide images (WSIs). (2) Methods: We recruited 504 eligible ccRCC patients from The Cancer Genome Atlas (TCGA) cohort and obtained 708 hematoxylin and eosin-stained WSIs for the development and internal validation of the SSL-CLAM model. Additionally, we obtained 445 WSIs from 188 ccRCC eligible patients in the Clinical Proteomic Tumor Analysis Consortium (CPTAC) cohort as an independent external validation set. A human-machine fusion approach was used to validate the added value of the SSL-CLAM model for pathologists. (3) Results: The SSL-CLAM model successfully diagnosed the five FG statuses (Grade-0, 1, 2, 3, and 4) of ccRCC, and achieved AUCs of 0.917 and 0.887 on the internal and external validation sets, respectively, outperforming a junior pathologist. For the normal/tumor classification (Grade-0, Grade-1/2/3/4) task, the SSL-CLAM model yielded AUCs close to 1 on both the internal and external validation sets. The SSL-CLAM model achieved a better performance for the two-tiered FG (Grade-0, Grade-1/2, and Grade-3/4) task, with AUCs of 0.936 and 0.915 on the internal and external validation sets, respectively. The human-machine diagnostic performance was superior to that of the SSL-CLAM model, showing promising prospects. In addition, the high-attention regions of the SSL-CLAM model showed that with an increasing FG status, the cell nuclei in the tumor region become larger, with irregular contours and increased cellular pleomorphism. (4) Conclusions: Our findings support the feasibility of using deep learning and human-machine fusion methods for FG classification on WSIs from ccRCC patients, which may assist pathologists in making diagnostic decisions.

Contribution of whole slide imaging-based deep learning in the assessment of intraoperative and postoperative sections in neuropathology

The pathological diagnosis of intracranial germinoma (IG), oligodendroglioma, and low-grade astrocytoma on intraoperative frozen section (IFS) and hematoxylin-eosin (HE)-staining section directly determines patients' treatment options, but it is a difficult task for pathologists. We aimed to investigate whether whole-slide imaging (WSI)-based deep learning can contribute new precision to the diagnosis of IG, oligodendroglioma, and low-grade astrocytoma. Two types of WSIs (500 IFSs and 832 HE-staining sections) were collected from 379 patients at multiple medical centers. Patients at Center 1 were split into the training, testing, and internal validation sets (3:1:1), while the other centers were the external validation sets. First, we subdivided WSIs into small tiles and selected tissue tiles using a tissue tile selection model. Then a tile-level classification model was established, and the majority voting method was used to determine the final diagnoses. Color jitter was applied to the tiles so that the deep learning (DL) models could adapt to the variations in the staining. Last, we investigated the effectiveness of model assistance. The internal validation accuracies of the IFS and HE models were 93.9% and 95.3%, respectively. The external validation accuracies of the IFS and HE models were 82.0% and 76.9%, respectively. Furthermore, the IFS and HE models can predict Ki-67 positive cell areas with R² of 0.81 and 0.86, respectively. With model assistance, the IFS and HE diagnosis accuracy of pathologists improved from 54.6%-69.7% and 53.5%-83.7% to 87.9%-93.9% and 86.0%-90.7%, respectively. Both the IFS model and the HE model can differentiate the three tumors, predict the expression of Ki-67, and improve the diagnostic accuracy of pathologists. The use of our model can assist clinicians in providing patients with optimal and timely treatment options.

Staging Systems and Nomograms for Soft Tissue Sarcoma

Reliable tools for prognosis prediction are crucially needed by oncologists so they can tailor individual treatments. However, the wide spectrum of histologies and prognostic behaviors of sarcomas challenges their development. In this field, nomograms could definitely better account for their granularity compared to the more widely used AJCC/UICC TNM staging system. Nomograms are predictive tools that incorporate multiple risk factors and return a numerical probability of a clinical event. Since the development of the first nomogram in 2002, several other nomograms have been built, either general, site-specific, histology-specific, or both. Recently, some new “dynamic” nomograms and prognostic tools have been developed, allowing doctors to “recalculate” a patient’s prognosis by taking into account the time since primary surgery, the event history, and the potential time-dependent effect of covariates. Due to these new tools, prognosis prediction is no longer limited to the time of the first computation but can be adapted and recalculated based on the occurrence (or not) of any event as time passes from the first computation. In this review, we aimed to give an overview of the available nomograms for STS and to help clinicians in the process of selecting the best tool for each patient.

Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer

Although it has long been known that the immune cell composition has a strong prognostic and predictive value in colorectal cancer (CRC), scoring systems such as the immunoscore (IS) or quantification of intraepithelial lymphocytes are only slowly being adopted into clinical routine use and have their limitations. To address this we established and evaluated a multistain deep learning model (MSDLM) utilizing artificial intelligence (AI) to determine the AImmunoscore (AIS) in more than 1,000 patients with CRC. Our model had high prognostic capabilities and outperformed other clinical, molecular and immune cell-based parameters. It could also be used to predict the response to neoadjuvant therapy in patients with rectal cancer. Using an explainable AI approach, we confirmed that the MSDLM's decisions were based on established cellular patterns of anti-tumor immunity. Hence, the AIS could provide clinicians with a valuable decision-making tool based on the tumor immune microenvironment.

Accurate Diagnosis and Survival Prediction of Bladder Cancer Using Deep Learning on Histological Slides

(1) Background: Early diagnosis and treatment are essential to reduce the mortality rate of bladder cancer (BLCA). We aimed to develop deep learning (DL)-based weakly supervised models for the diagnosis of BLCA and prediction of overall survival (OS) in muscle-invasive bladder cancer (MIBC) patients using whole slide digitized histological images (WSIs). (2) Methods: Diagnostic and prognostic models were developed using 926 WSIs of 412 BLCA patients from The Cancer Genome Atlas cohort. We collected 250 WSIs of 150 BLCA patients from the Renmin Hospital of Wuhan University cohort for external validation of the models. Two DL models were developed: a BLCA diagnostic model (named BlcaMIL) and an MIBC prognostic model (named MibcMLP). (3) Results: The BlcaMIL model identified BLCA with accuracy 0.987 in the external validation set, comparable to that of expert uropathologists and outperforming a junior pathologist. The C-index values for the MibcMLP model on the internal and external validation sets were 0.631 and 0.622, respectively. The risk score predicted by MibcMLP was a strong predictor independent of existing clinical or histopathologic indicators, as demonstrated by univariate Cox (HR = 2.390, p < 0.0001) and multivariate Cox (HR = 2.414, p < 0.0001) analyses. The interpretability of DL models can help in the analysis of critical regions associated with tumors to enrich the information obtained from WSIs. Furthermore, the expression of six genes (ANAPC7, MAPKAPK5, COX19, LINC01106, AL161431.1 and MYO16-AS1) was significantly associated with MibcMLP-predicted risk scores, revealing possible potential biological correlations. (4) Conclusions: Our study developed DL models for accurately diagnosing BLCA and predicting OS in MIBC patients, which will help promote the precise pathological diagnosis of BLCA and risk stratification of MIBC to improve clinical treatment decisions.

Evaluation of deep learning techniques for identification of sarcoma-causing carcinogenic mutations

The abnormal growth of human healthy cells is called cancer. One of the major types of cancer is sarcoma, mostly found in human bones and soft tissue cells. It commonly occurs in children. According to a survey of the United States of America, there are more than 17,000 sarcoma patients registered each year which is 15% of all cancer cases. Recognition of cancer at its early stage saves many lives. The proposed study developed a framework for the early detection of human sarcoma cancer using deep learning Recurrent Neural Network (RNN) algorithms. The DNA of a human cell is made up of 25,000 to 30,000 genes. Each gene is represented by sequences of nucleotides. The nucleotides in a sequence of a driver gene can change which is termed as mutations. Some mutations can cause cancer. There are seven types of a gene whose mutation causes sarcoma cancer. The study uses the dataset which has been taken from more than 134 samples and includes 141 mutations in 8 driver genes. On these gene sequences RNN algorithms Long and Short-Term Memory (LSTM), Gated Recurrent Units and Bi-directional LSTM (Bi-LSTM) are used for training. Rigorous testing techniques such as Self-consistency testing, independent set testing, 10-fold cross-validation test are applied for the validation of results. These validation techniques yield several metrics such as Area Under the Curve (AUC), sensitivity, specificity, Mathew's correlation coefficient, loss, and accuracy. The proposed algorithm exhibits an accuracy of 99.6% with an AUC value of 1.00.

The diagnosis, classification, and treatment of sarcoma in this era of artificial intelligence and immunotherapy

Soft-tissue sarcomas (STS) represent a group of rare and heterogeneous tumors associated with several challenges, including incorrect or late diagnosis, the lack of clinical expertise, and limited therapeutic options. Digital pathology and radiomics represent transformative technologies that appear promising for improving the accuracy of cancer diagnosis, characterization and monitoring. Herein, we review the potential role of the application of digital pathology and radiomics in managing patients with STS. We have particularly described the main results and the limits of the studies using radiomics to refine diagnosis or predict the outcome of patients with soft-tissue sarcomas. We also discussed the current limitation of implementing radiomics in routine settings. Standard management approaches for STS have not improved since the early 1970s. Immunotherapy has revolutionized cancer treatment; nonetheless, immuno-oncology agents have not yet been approved for patients with STS. However, several lines of evidence indicate that immunotherapy may represent an efficient therapeutic strategy for this group of diseases. Thus, we emphasized the remarkable potential of immunotherapy in sarcoma treatment by focusing on recent data regarding the immune landscape of these tumors. We have particularly emphasized the fact that the development of immunotherapy for sarcomas is not an aspect of histology (except for alveolar soft-part sarcoma) but rather that of the tumor microenvironment. Future studies investigating immunotherapy strategies in sarcomas should incorporate at least the presence of tertiary lymphoid structures as a stratification factor in their design, besides including a strong translational program that will allow for a better understanding of the determinants involved in sensitivity and treatment resistance to immune-oncology agents.

Multi-modality artificial intelligence in digital pathology

In common medical procedures, the time-consuming and expensive nature of obtaining test results plagues doctors and patients. Digital pathology research allows using computational technologies to manage data, presenting an opportunity to improve the efficiency of diagnosis and treatment. Artificial intelligence (AI) has a great advantage in the data analytics phase. Extensive research has shown that AI algorithms can produce more up-to-date and standardized conclusions for whole slide images. In conjunction with the development of high-throughput sequencing technologies, algorithms can integrate and analyze data from multiple modalities to explore the correspondence between morphological features and gene expression. This review investigates using the most popular image data, hematoxylin-eosin stained tissue slide images, to find a strategic solution for the imbalance of healthcare resources. The article focuses on the role that the development of deep learning technology has in assisting doctors' work and discusses the opportunities and challenges of AI.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

Emerging Applications of Deep Learning in Bone Tumors: Current Advances and Challenges

Deep learning is a subfield of state-of-the-art artificial intelligence (AI) technology, and multiple deep learning-based AI models have been applied to musculoskeletal diseases. Deep learning has shown the capability to assist clinical diagnosis and prognosis prediction in a spectrum of musculoskeletal disorders, including fracture detection, cartilage and spinal lesions identification, and osteoarthritis severity assessment. Meanwhile, deep learning has also been extensively explored in diverse tumors such as prostate, breast, and lung cancers. Recently, the application of deep learning emerges in bone tumors. A growing number of deep learning models have demonstrated good performance in detection, segmentation, classification, volume calculation, grading, and assessment of tumor necrosis rate in primary and metastatic bone tumors based on both radiological (such as X-ray, CT, MRI, SPECT) and pathological images, implicating a potential for diagnosis assistance and prognosis prediction of deep learning in bone tumors. In this review, we first summarized the workflows of deep learning methods in medical images and the current applications of deep learning-based AI for diagnosis and prognosis prediction in bone tumors. Moreover, the current challenges in the implementation of the deep learning method and future perspectives in this field were extensively discussed.

Sarcoma Common MHC-I Haplotype Restricts Tumor-Specific CD8+ T Cell Response

Simple Summary

Immunotherapy targeting immune checkpoint pathways have recently attracted great attention in cancer treatment, but better strategies are needed to identify patients likely to benefit from it. The major histocompatibility complex (MHC) class I expression in cancer cells greatly influences the outcome of T cell-mediated immunotherapy. Here, we determined the prevalent HLA class I allelic variants in a sarcoma population. We characterized patient CD8+ T-cells and demonstrated low cytolysis to autologous tumor cells. Moreover, we used a co-culture model of autologous T-cells and PD-L1-deficient or positive biopsies of rare sarcomas to determine whether HLA-I influences tumor survival.

Abstract

The major histocompatibility complex (MHC) class I expression in cancer cells has a crucial impact on the outcome of T cell-mediated cancer immunotherapy. We now determined the HLA class I allelic variants and their expression in PD-L1-deficient and positive rare sarcoma tissues. Tumor tissues were HLA-I classified based on HLA-A and -B alleles, and for class II, the HLA-DR-B by Taqman genomic PCRs. The HLA-A24*:10-B73*:01 haplotype was the most common. A general down-regulation or deletion of HLA-B mRNA and HLA-A was observed, compared to HLA-DR-B. HLA-I was almost too low to be detectable by immunohistochemistry and 32% of grade III cases were positive to PD-L1. Functional cytotoxic assays co-culturing patient biopsies with autologous T cells were used to assess their ability to kill matched tumor cells. These results establish that deletion of HLA-I loci together with their down-regulation in individual patient restrict the autologous lymphocyte cytotoxic activity, even in the presence of the immune checkpoint blocking antibody, Nivolumab. Additionally, the proposed cytotoxic test suggests a strategy to assess the sensitivity of tumor cells to T cell-mediated attack at the level of the individual patient.

Introduction to supervised machine learning in clinical epidemiology

Machine learning refers to a series of processes in which a computer finds rules from a vast amount of data. With recent advances in computer technology and the availability of a wide variety of health data, machine learning has rapidly developed and been applied in medical research. Currently, there are three types of machine learning: supervised, unsupervised, and reinforcement learning. In medical research, supervised learning is commonly used for diagnoses and prognoses, while unsupervised learning is used for phenotyping a disease, and reinforcement learning for maximizing favorable results, such as optimization of total patients' waiting time in the emergency department. The present article focuses on the concept and application of supervised learning in medicine, the most commonly used machine learning approach in medicine, and provides a brief explanation of four algorithms widely used for prediction (random forests, gradient-boosted decision tree, support vector machine, and neural network). Among these algorithms, the neural network has further developed into deep learning algorithms to solve more complex tasks. Along with simple classification problems, deep learning is commonly used to process medical imaging, such as retinal fundus photographs for diabetic retinopathy diagnosis. Although machine learning can bring new insights into medicine by processing a vast amount of data that are often beyond human capacity, algorithms can also fail when domain knowledge is neglected. The combination of algorithms and human cognitive ability is a key to the successful application of machine learning in medicine.

Predicting the Risk of Dental Implant Loss Using Deep Learning

AIM

To investigate the feasibility of predicting dental implant loss risk with deep learning (DL) based on preoperative cone-beam computed tomography.

MATERIALS AND METHODS

Six hundred and three patients who underwent implant surgery (279 high-risk patients who did and 324 low-risk patients who did not experience implant loss within 5 years) from January 2012 to January 2020 were enrolled. Three models, a logistic regression clinical model (CM) based on clinical features, a DL model based on radiography features, and an integrated model (IM) developed by combining CM with DL, were developed to predict the 5-year implant loss risk. The area under the receiver operating characteristic curve (AUC) was used to evaluate the model performance. Time to implant loss was considered for both groups, and Kaplan-Meier curves were created and compared by the log-rank test.

RESULTS

The IM exhibited the best performance in predicting implant loss risk [AUC = 0.90, 95% confidence interval (CI) 0.84-0.95], followed by the DL model (AUC = 0.87, 95% CI 0.80-0.92) and the CM (AUC = 0.72, 95% CI 0.63-0.79).

CONCLUSION

Our study offers preliminary evidence that both the DL model and IM performed well in predicting implant fate within 5 years and thus may greatly facilitate implant practitioners in assessing preoperative risks.

Development of a Deep Learning Model for Malignant Small Bowel Tumors Survival: A SEER-Based Study

Background This study aims to explore a deep learning (DL) algorithm for developing a prognostic model and perform survival analyses in SBT patients. Methods The demographic and clinical features of patients with SBTs were extracted from the Surveillance, Epidemiology and End Results (SEER) database. We randomly split the samples into the training set and the validation set at 7:3. Cox proportional hazards (Cox-PH) analysis and the DeepSurv algorithm were used to develop models. The performance of the Cox-PH and DeepSurv models was evaluated using receiver operating characteristic curves, calibration curves, C-statistics and decision-curve analysis (DCA). A Kaplan−Meier (K−M) survival analysis was performed for further explanation on prognostic effect of the Cox-PH model. Results The multivariate analysis demonstrated that seven variables were associated with cancer-specific survival (CSS) (all p < 0.05). The DeepSurv model showed better performance than the Cox-PH model (C-index: 0.871 vs. 0.866). The calibration curves and DCA revealed that the two models had good discrimination and calibration. Moreover, patients with ileac malignancy and N2 stage disease were not responding to surgery according to the K−M analysis. Conclusions This study reported a DeepSurv model that performed well in CSS in SBT patients. It might offer insights into future research to explore more DL algorithms in cohort studies.

Artificial intelligence to identify genetic alterations in conventional histopathology

Precision oncology relies on the identification of targetable molecular alterations in tumor tissues. In many tumor types, a limited set of molecular tests is currently part of standard diagnostic workflows. However, universal testing for all targetable alterations, especially rare ones, is limited by the cost and availability of molecular assays. From 2017 to 2021, multiple studies have shown that artificial intelligence (AI) methods can predict the probability of specific genetic alterations directly from conventional hematoxylin and eosin (H&E) tissue slides. Although these methods are currently less accurate than gold-standard testing (e.g. immunohistochemistry, polymerase chain reaction or next-generation sequencing), they could be used as pre-screening tools to reduce the workload of genetic analyses. In this systematic literature review, we summarize the state of the art in predicting molecular alterations from H&E using AI. We found that AI methods perform reasonably well across multiple tumor types, although few algorithms have been broadly validated. In addition, we found that genetic alterations in FGFR, IDH, PIK3CA, BRAF, TP53 and DNA repair pathways are predictable from H&E in multiple tumor types, while many other genetic alterations have rarely been investigated or were only poorly predictable. Finally, we discuss the next steps for the implementation of AI-based surrogate tests in diagnostic workflows. This article is protected by copyright. All rights reserved.

Immunotherapy for SMARCB1-Deficient Sarcomas: Current Evidence and Future Developments

Mutations in subunits of the SWItch Sucrose Non-Fermentable (SWI/SNF) complex occur in 20% of all human tumors. Among these, the core subunit SMARCB1 is the most frequently mutated, and SMARCB1 loss represents a founder driver event in several malignancies, such as malignant rhabdoid tumors (MRT), epithelioid sarcoma, poorly differentiated chordoma, and renal medullary carcinoma (RMC). Intriguingly, SMARCB1-deficient pediatric MRT and RMC have recently been reported to be immunogenic, despite their very simple genome and low tumor mutational burden. Responses to immune checkpoint inhibitors have further been reported in some SMARCB1-deficient diseases. Here, we will review the preclinical data and clinical data that suggest that immunotherapy, including immune checkpoint inhibitors, may represent a promising therapeutic strategy for SMARCB1-defective tumors. We notably discuss the heterogeneity that exists among the spectrum of malignancies driven by SMARCB1-loss, and highlight challenges that are at stake for developing a personalized immunotherapy for these tumors, notably using molecular profiling of the tumor and of its microenvironment.

Harnessing big data to characterize immune-related adverse events

Immune-checkpoint inhibitors (ICIs) have transformed patient care in oncology but are associated with a unique spectrum of organ-specific inflammatory toxicities known as immune-related adverse events (irAEs). Given the expanding use of ICIs, an increasing number of patients with cancer experience irAEs, including severe irAEs. Proper diagnosis and management of irAEs are important to optimize the quality of life and long-term outcomes of patients receiving ICIs; however, owing to the substantial heterogeneity within irAEs, and despite multicentre initiatives, performing clinical studies of these toxicities with a sufficient cohort size is challenging. Pioneering studies from the past few years have demonstrated that aggregate clinical data, real-world data (such as data on pharmacovigilance or from electronic health records) and multi-omics data are alternative tools well suited to investigating the underlying mechanisms and clinical presentations of irAEs. In this Perspective, we summarize the advantages and shortcomings of different sources of 'big data' for the study of irAEs and highlight progress made using such data to identify biomarkers of irAE risk, evaluate associations between irAEs and therapeutic efficacy, and characterize the effects of demographic and anthropometric factors on irAE risk. Harnessing big data will accelerate research on irAEs and provide key insights that will improve the clinical management of patients receiving ICIs.

Multimodal Deep Learning for Prognosis Prediction in Renal Cancer

BACKGROUND

Clear-cell renal cell carcinoma (ccRCC) is common and associated with substantial mortality. TNM stage and histopathological grading have been the sole determinants of a patient's prognosis for decades and there are few prognostic biomarkers used in clinical routine. Management of ccRCC involves multiple disciplines such as urology, radiology, oncology, and pathology and each of these specialties generates highly complex medical data. Here, artificial intelligence (AI) could prove extremely powerful to extract meaningful information to benefit patients.

OBJECTIVE

In the study, we developed and evaluated a multimodal deep learning model (MMDLM) for prognosis prediction in ccRCC.

DESIGN SETTING AND PARTICIPANTS

Two mixed cohorts of non-metastatic and metastatic ccRCC patients were used: (1) The Cancer Genome Atlas cohort including 230 patients and (2) the Mainz cohort including 18 patients with ccRCC. For each of these patients, we trained the MMDLM on multiscale histopathological images, CT/MRI scans, and genomic data from whole exome sequencing.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

Outcome measurements included Harrell's concordance index (C-index) and also various performance parameters for predicting the 5-year survival status (5YSS). Different visualization techniques were used to make our model more transparent.

RESULTS

The MMDLM showed great performance in predicting the prognosis of ccRCC patients with a mean C-index of 0.7791 and a mean accuracy of 83.43%. Training on a combination of data from different sources yielded significantly better results compared to when only one source was used. Furthermore, the MMDLM's prediction was an independent prognostic factor outperforming other clinical parameters.

INTERPRETATION

Multimodal deep learning can contribute to prognosis prediction in ccRCC and potentially help to improve the clinical management of this disease.

PATIENT SUMMARY

An AI-based computer program can analyze various medical data (microscopic images, CT/MRI scans, and genomic data) simultaneously and thereby predict the survival time of patients with renal cancer.

Weakly supervised annotation-free cancer detection and prediction of genotype in routine histopathology

Deep learning is a powerful tool in computational pathology: it can be used for tumor detection and for predicting genetic alterations based on histopathology images alone. Conventionally, tumor detection and prediction of genetic alterations are two separate workflows. Newer methods have combined them, but require complex, manually engineered computational pipelines, restricting reproducibility and robustness. To address these issues, we present a new method for simultaneous tumor detection and prediction of genetic alterations: The Slide-Level Assessment Model (SLAM) uses a single off-the-shelf neural network to predict molecular alterations directly from routine pathology slides without any manual annotations, improving upon previous methods by automatically excluding normal and non-informative tissue regions. SLAM requires only standard programming libraries and is conceptually simpler than previous approaches. We have extensively validated SLAM for clinically relevant tasks using two large multicentric cohorts of colorectal cancer patients, Darmkrebs: Chancen der Verhütung durch Screening (DACHS) from Germany and Yorkshire Cancer Research Bowel Cancer Improvement Programme (YCR-BCIP) from the UK. We show that SLAM yields reliable slide-level classification of tumor presence with an area under the receiver operating curve (AUROC) of 0.980 (confidence interval 0.975, 0.984; n = 2,297 tumor and n = 1,281 normal slides). In addition, SLAM can detect microsatellite instability (MSI)/mismatch repair deficiency (dMMR) or microsatellite stability/mismatch repair proficiency with an AUROC of 0.909 (0.888, 0.929; n = 2,039 patients) and BRAF mutational status with an AUROC of 0.821 (0.786, 0.852; n = 2,075 patients). The improvement with respect to previous methods was validated in a large external testing cohort in which MSI/dMMR status was detected with an AUROC of 0.900 (0.864, 0.931; n = 805 patients). In addition, SLAM provides human-interpretable visualization maps, enabling the analysis of multiplexed network predictions by human experts. In summary, SLAM is a new simple and powerful method for computational pathology that could be applied to multiple disease contexts. © 2021 The Authors. The Journal of Pathology published by John Wiley & Sons, Ltd. on behalf of The Pathological Society of Great Britain and Ireland.