Domain Adaptation-Based Deep Learning for Automated Tumor Cell (TC) Scoring and Survival Analysis on PD-L1 Stained Tissue Images

Leveraging immuno-fluorescence data to reduce pathologist annotation requirements in lung tumor segmentation using deep learning

The main bottleneck in training a robust tumor segmentation algorithm for non-small cell lung cancer (NSCLC) on H&E is generating sufficient ground truth annotations. Various approaches for generating tumor labels to train a tumor segmentation model was explored. A large dataset of low-cost low-accuracy panCK-based annotations was used to pre-train the model and determine the minimum required size of the expensive but highly accurate pathologist annotations dataset. PanCK pre-training was compared to foundation models and various architectures were explored for model backbone. Proper study design and sample procurement for training a generalizable model that captured variations in NSCLC H&E was studied. H&E imaging was performed on 112 samples (three centers, two scanner types, different staining and imaging protocols). Attention U-Net architecture was trained using the large panCK-based annotations dataset (68 samples, total area 10,326 [mm2]) followed by fine-tuning using a small pathologist annotations dataset (80 samples, total area 246 [mm2]). This approach resulted in mean intersection over union (mIoU) of 82% [77 87]. Using panCK pretraining provided better performance compared to foundation models and allowed for 70% reduction in pathologist annotations with no drop in performance. Study design ensured model generalizability over variations on H&E where performance was consistent across centers, scanners, and subtypes.

Artificial intelligence enhances whole-slide interpretation of PD-L1 CPS in triple-negative breast cancer: A multi-institutional ring study

BACKGROUND AND AIMS

Evaluation of the programmed cell death ligand-1 (PD-L1) combined positive score (CPS) is vital to predict the efficacy of the immunotherapy in triple-negative breast cancer (TNBC), but pathologists show substantial variability in the consistency and accuracy of the interpretation. It is of great importance to establish an objective and effective method which is highly repeatable.

METHODS

We proposed a model in a deep learning-based framework, which at the patch level incorporated cell analysis and tissue region analysis, followed by the whole-slide level fusion of patch results. Three rounds of ring studies (RSs) were conducted. Twenty-one pathologists of different levels from four institutions evaluated the PD-L1 CPS in TNBC specimens as continuous scores by visual assessment and our artificial intelligence (AI)-assisted method.

RESULTS

In the visual assessment, the interpretation results of PD-L1 (Dako 22C3) CPS by different levels of pathologists have significant differences and showed weak consistency. Using AI-assisted interpretation, there were no significant differences between all pathologists (P = 0.43), and the intraclass correlation coefficient (ICC) value was increased from 0.618 [95% confidence interval (CI) = 0.524-0.719] to 0.931 (95% CI = 0.902-0.955). The accuracy of interpretation result is further improved to 0.919 (95% CI = 0.886-0.947). Acceptance of AI results by junior pathologists was the highest among all levels, and 80% of the AI results were accepted overall.

CONCLUSION

With the help of the AI-assisted diagnostic method, different levels of pathologists achieved excellent consistency and repeatability in the interpretation of PD-L1 (Dako 22C3) CPS. Our AI-assisted diagnostic approach was proved to strengthen the consistency and repeatability in clinical practice.

Cross-site prognosis prediction for nasopharyngeal carcinoma from incomplete multi-modal data

Accurate prognosis prediction for nasopharyngeal carcinoma based on magnetic resonance (MR) images assists in the guidance of treatment intensity, thus reducing the risk of recurrence and death. To reduce repeated labor and sufficiently explore domain knowledge, aggregating labeled/annotated data from external sites enables us to train an intelligent model for a clinical site with unlabeled data. However, this task suffers from the challenges of incomplete multi-modal examination data fusion and image data heterogeneity among sites. This paper proposes a cross-site survival analysis method for prognosis prediction of nasopharyngeal carcinoma from domain adaptation viewpoint. Utilizing a Cox model as the basic framework, our method equips it with a cross-attention based multi-modal fusion regularization. This regularization model effectively fuses the multi-modal information from multi-parametric MR images and clinical features onto a domain-adaptive space, despite the absence of some modalities. To enhance the feature discrimination, we also extend the contrastive learning technique to censored data cases. Compared with the conventional approaches which directly deploy a trained survival model in a new site, our method achieves superior prognosis prediction performance in cross-site validation experiments. These results highlight the key role of cross-site adaptability of our method and support its value in clinical practice.

Artificial intelligence-based assessment of PD-L1 expression in diffuse large B cell lymphoma

Diffuse large B cell lymphoma (DLBCL) is an aggressive blood cancer known for its rapid progression and high incidence. The growing use of immunohistochemistry (IHC) has significantly contributed to the detailed cell characterization, thereby playing a crucial role in guiding treatment strategies for DLBCL. In this study, we developed an AI-based image analysis approach for assessing PD-L1 expression in DLBCL patients. PD-L1 expression represents as a major biomarker for screening patients who can benefit from targeted immunotherapy interventions. In particular, we performed large-scale cell annotations in IHC slides, encompassing over 5101 tissue regions and 146,439 live cells. Extensive experiments in primary and validation cohorts demonstrated the defined quantitative rule helped overcome the difficulty of identifying specific cell types. In assessing data obtained from fine needle biopsies, experiments revealed that there was a higher level of agreement in the quantitative results between Artificial Intelligence (AI) algorithms and pathologists, as well as among pathologists themselves, in comparison to the data obtained from surgical specimens. We highlight that the AI-enabled analytics enhance the objectivity and interpretability of PD-L1 quantification to improve the targeted immunotherapy development in DLBCL patients.

Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images

Programmed death-ligand 1 (PD-L1) expression is currently used in the clinic to assess eligibility for immune-checkpoint inhibitors via the tumor proportion score (TPS), but its efficacy is limited by high interobserver variability. Multiple papers have presented systems for the automatic quantification of TPS, but none report on the task of determining cell-level PD-L1 expression and often reserve their evaluation to a single PD-L1 monoclonal antibody or clinical center. In this paper, we report on a deep learning algorithm for detecting PD-L1 negative and positive tumor cells at a cellular level and evaluate it on a cell-level reference standard established by six readers on a multi-centric, multi PD-L1 assay dataset. This reference standard also provides for the first time a benchmark for computer vision algorithms. In addition, in line with other papers, we also evaluate our algorithm at slide-level by measuring the agreement between the algorithm and six pathologists on TPS quantification. We find a moderately low interobserver agreement at cell-level level (mean reader-reader F1 score = 0.68) which our algorithm sits slightly under (mean reader-AI F1 score = 0.55), especially for cases from the clinical center not included in the training set. Despite this, we find good AI-pathologist agreement on quantifying TPS compared to the interobserver agreement (mean reader-reader Cohen's kappa = 0.54, 95% CI 0.26-0.81, mean reader-AI kappa = 0.49, 95% CI 0.27-0.72). In conclusion, our deep learning algorithm demonstrates promise in detecting PD-L1 expression at a cellular level and exhibits favorable agreement with pathologists in quantifying the tumor proportion score (TPS). We publicly release our models for use via the Grand-Challenge platform.

Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives

Deep learning has demonstrated remarkable performance across various tasks in medical imaging. However, these approaches primarily focus on supervised learning, assuming that the training and testing data are drawn from the same distribution. Unfortunately, this assumption may not always hold true in practice. To address these issues, unsupervised domain adaptation (UDA) techniques have been developed to transfer knowledge from a labeled domain to a related but unlabeled domain. In recent years, significant advancements have been made in UDA, resulting in a wide range of methodologies, including feature alignment, image translation, self-supervision, and disentangled representation methods, among others. In this paper, we provide a comprehensive literature review of recent deep UDA approaches in medical imaging from a technical perspective. Specifically, we categorize current UDA research in medical imaging into six groups and further divide them into finer subcategories based on the different tasks they perform. We also discuss the respective datasets used in the studies to assess the divergence between the different domains. Finally, we discuss emerging areas and provide insights and discussions on future research directions to conclude this survey.

Weakly Supervised Deep Learning Predicts Immunotherapy Response in Solid Tumors Based on PD-L1 Expression

Programmed death-ligand 1 (PD-L1) IHC is the most commonly used biomarker for immunotherapy response. However, quantification of PD-L1 status in pathology slides is challenging. Neither manual quantification nor a computer-based mimicking of manual readouts is perfectly reproducible, and the predictive performance of both approaches regarding immunotherapy response is limited. In this study, we developed a deep learning (DL) method to predict PD-L1 status directly from raw IHC image data, without explicit intermediary steps such as cell detection or pigment quantification. We trained the weakly supervised model on PD-L1-stained slides from the non-small cell lung cancer (NSCLC)-Memorial Sloan Kettering (MSK) cohort (N = 233) and validated it on the pan-cancer-Vall d'Hebron Institute of Oncology (VHIO) cohort (N = 108). We also investigated the performance of the model to predict response to immune checkpoint inhibitors (ICI) in terms of progression-free survival. In the pan-cancer-VHIO cohort, the performance was compared with tumor proportion score (TPS) and combined positive score (CPS). The DL model showed good performance in predicting PD-L1 expression (TPS ≥ 1%) in both NSCLC-MSK and pan-cancer-VHIO cohort (AUC 0.88 ± 0.06 and 0.80 ± 0.03, respectively). The predicted PD-L1 status showed an improved association with response to ICIs [HR: 1.5 (95% confidence interval: 1-2.3), P = 0.049] compared with TPS [HR: 1.4 (0.96-2.2), P = 0.082] and CPS [HR: 1.2 (0.79-1.9), P = 0.386]. Notably, our explainability analysis showed that the model does not just look at the amount of brown pigment in the IHC slides, but also considers morphologic factors such as lymphocyte conglomerates. Overall, end-to-end weakly supervised DL shows potential for improving patient stratification for cancer immunotherapy by analyzing PD-L1 IHC, holistically integrating morphology and PD-L1 staining intensity.

SIGNIFICANCE

The weakly supervised DL model to predict PD-L1 status from raw IHC data, integrating tumor staining intensity and morphology, enables enhanced patient stratification in cancer immunotherapy compared with traditional pathologist assessment.

Predicting Lymph Node Metastasis From Primary Cervical Squamous Cell Carcinoma Based on Deep Learning in Histopathologic Images

We developed a deep learning framework to accurately predict the lymph node status of patients with cervical cancer based on hematoxylin and eosin-stained pathological sections of the primary tumor. In total, 1524 hematoxylin and eosin-stained whole slide images (WSIs) of primary cervical tumors from 564 patients were used in this retrospective, proof-of-concept study. Primary tumor sections (1161 WSIs) were obtained from 405 patients who underwent radical cervical cancer surgery at the Fudan University Shanghai Cancer Center (FUSCC) between 2008 and 2014; 165 and 240 patients were negative and positive for lymph node metastasis, respectively (including 166 with positive pelvic lymph nodes alone and 74 with positive pelvic and para-aortic lymph nodes). We constructed and trained a multi-instance deep convolutional neural network based on a multiscale attention mechanism, in which an internal independent test set (100 patients, 228 WSIs) from the FUSCC cohort and an external independent test set (159 patients, 363 WSIs) from the Cervical Squamous Cell Carcinoma and Endocervical Adenocarcinoma cohort of the Cancer Genome Atlas program database were used to evaluate the predictive performance of the network. In predicting the occurrence of lymph node metastasis, our network achieved areas under the receiver operating characteristic curve of 0.87 in the cross-validation set, 0.84 in the internal independent test set of the FUSCC cohort, and 0.75 in the external test set of the Cervical Squamous Cell Carcinoma and Endocervical Adenocarcinoma cohort of the Cancer Genome Atlas program. For patients with positive pelvic lymph node metastases, we retrained the network to predict whether they also had para-aortic lymph node metastases. Our network achieved areas under the receiver operating characteristic curve of 0.91 in the cross-validation set and 0.88 in the test set of the FUSCC cohort. Deep learning analysis based on pathological images of primary foci is very likely to provide new ideas for preoperatively assessing cervical cancer lymph node status; its true value must be validated with cervical biopsy specimens and large multicenter datasets.

Novel tools for early diagnosis and precision treatment based on artificial intelligence

Lung cancer has the highest mortality rate among all cancers in the world. Hence, early diagnosis and personalized treatment plans are crucial to improving its 5-year survival rate. Chest computed tomography (CT) serves as an essential tool for lung cancer screening, and pathology images are the gold standard for lung cancer diagnosis. However, medical image evaluation relies on manual labor and suffers from missed diagnosis or misdiagnosis, and physician heterogeneity. The rapid development of artificial intelligence (AI) has brought a whole novel opportunity for medical task processing, demonstrating the potential for clinical application in lung cancer diagnosis and treatment. AI technologies, including machine learning and deep learning, have been deployed extensively for lung nodule detection, benign and malignant classification, and subtype identification based on CT images. Furthermore, AI plays a role in the non-invasive prediction of genetic mutations and molecular status to provide the optimal treatment regimen, and applies to the assessment of therapeutic efficacy and prognosis of lung cancer patients, enabling precision medicine to become a reality. Meanwhile, histology-based AI models assist pathologists in typing, molecular characterization, and prognosis prediction to enhance the efficiency of diagnosis and treatment. However, the leap to extensive clinical application still faces various challenges, such as data sharing, standardized label acquisition, clinical application regulation, and multimodal integration. Nevertheless, AI holds promising potential in the field of lung cancer to improve cancer care.

Deep Learning for Lung Cancer Diagnosis, Prognosis and Prediction Using Histological and Cytological Images: A Systematic Review

Lung cancer is one of the deadliest cancers worldwide, with a high incidence rate, especially in tobacco smokers. Lung cancer accurate diagnosis is based on distinct histological patterns combined with molecular data for personalized treatment. Precise lung cancer classification from a single H&E slide can be challenging for a pathologist, requiring most of the time additional histochemical and special immunohistochemical stains for the final pathology report. According to WHO, small biopsy and cytology specimens are the available materials for about 70% of lung cancer patients with advanced-stage unresectable disease. Thus, the limited available diagnostic material necessitates its optimal management and processing for the completion of diagnosis and predictive testing according to the published guidelines. During the new era of Digital Pathology, Deep Learning offers the potential for lung cancer interpretation to assist pathologists' routine practice. Herein, we systematically review the current Artificial Intelligence-based approaches using histological and cytological images of lung cancer. Most of the published literature centered on the distinction between lung adenocarcinoma, lung squamous cell carcinoma, and small cell lung carcinoma, reflecting the realistic pathologist's routine. Furthermore, several studies developed algorithms for lung adenocarcinoma predominant architectural pattern determination, prognosis prediction, mutational status characterization, and PD-L1 expression status estimation.

Artificial intelligence for prediction of response to cancer immunotherapy

Artificial intelligence (AI) indicates the application of machines to imitate intelligent behaviors for solving complex tasks with minimal human intervention, including machine learning and deep learning. The use of AI in medicine improves health-care systems in multiple areas such as diagnostic confirmation, risk stratification, analysis, prognosis prediction, treatment surveillance, and virtual health support, which has considerable potential to revolutionize and reshape medicine. In terms of immunotherapy, AI has been applied to unlock underlying immune signatures to associate with responses to immunotherapy indirectly as well as predict responses to immunotherapy responses directly. The AI-based analysis of high-throughput sequences and medical images can provide useful information for management of cancer immunotherapy considering the excellent abilities in selecting appropriate subjects, improving therapeutic regimens, and predicting individualized prognosis. In present review, we aim to evaluate a broad framework about AI-based computational approaches for prediction of response to cancer immunotherapy on both indirect and direct manners. Furthermore, we summarize our perspectives about challenges and opportunities of further AI applications on cancer immunotherapy relating to clinical practicability.

Artificial intelligence-based prediction of clinical outcome in immunotherapy and targeted therapy of lung cancer

Lung cancer accounts for the main proportion of malignancy-related deaths and most patients are diagnosed at an advanced stage. Immunotherapy and targeted therapy have great advances in application in clinics to treat lung cancer patients, yet the efficacy is unstable. The response rate of these therapies varies among patients. Some biomarkers have been proposed to predict the outcomes of immunotherapy and targeted therapy, including programmed cell death-ligand 1 (PD-L1) expression and oncogene mutations. Nevertheless, the detection tests are invasive, time-consuming, and have high demands on tumor tissue. The predictive performance of conventional biomarkers is also unsatisfactory. Therefore, novel biomarkers are needed to effectively predict the outcomes of immunotherapy and targeted therapy. The application of artificial intelligence (AI) can be a possible solution, as it has several advantages. AI can help identify features that are unable to be used by humans and perform repetitive tasks. By combining AI methods with radiomics, pathology, genomics, transcriptomics, proteomics, and clinical data, the integrated model has shown predictive value in immunotherapy and targeted therapy, which significantly improves the precision treatment of lung cancer patients. Herein, we reviewed the application of AI in predicting the outcomes of immunotherapy and targeted therapy in lung cancer patients, and discussed the challenges and future directions in this field.

The Role of Pathology-Based Methods in Qualitative and Quantitative Approaches to Cancer Immunotherapy

Simple Summary

Immunotherapy has become the filar of modern oncological treatment, and programmed death-ligand 1 expression is one of the primary immune markers assessed by pathologists. However, there are still some issues concerning the evaluation of the marker and limited information about the interaction between the tumour and associated immune cells. Recent studies have focused on cancer immunology to try to understand the complex tumour microenvironment, and multiplex imaging methods are more widely used for this purpose. The presented article aims to provide an overall review of a different multiplex in situ method using spectral imaging, supported by automated image-acquisition and software-assisted marker visualisation and interpretation. Multiplex imaging methods could improve the current understanding of complex tumour-microenvironment immunology and could probably help to better match patients to appropriate treatment regimens.

Abstract

Immune checkpoint inhibitors, including those concerning programmed cell death 1 (PD-1) and its ligand (PD-L1), have revolutionised the cancer therapy approach in the past decade. However, not all patients benefit from immunotherapy equally. The prediction of patient response to this type of therapy is mainly based on conventional immunohistochemistry, which is limited by intraobserver variability, semiquantitative assessment, or single-marker-per-slide evaluation. Multiplex imaging techniques and digital image analysis are powerful tools that could overcome some issues concerning tumour-microenvironment studies. This novel approach to biomarker assessment offers a better understanding of the complicated interactions between tumour cells and their environment. Multiplex labelling enables the detection of multiple markers simultaneously and the exploration of their spatial organisation. Evaluating a variety of immune cell phenotypes and differentiating their subpopulations is possible while preserving tissue histology in most cases. Multiplexing supported by digital pathology could allow pathologists to visualise and understand every cell in a single tissue slide and provide meaning in a complex tumour-microenvironment contexture. This review aims to provide an overview of the different multiplex imaging methods and their application in PD-L1 biomarker assessment. Moreover, we discuss digital imaging techniques, with a focus on slide scanners and software.

Automated PD-L1 Scoring Using Artificial Intelligence in Head and Neck Squamous Cell Carcinoma

Immune checkpoint inhibitors (ICI) represent a new therapeutic approach in recurrent and metastatic head and neck squamous cell carcinoma (HNSCC). The patient selection for the PD-1/PD-L1 inhibitor therapy is based on the degree of PD-L1 expression in immunohistochemistry reflected by manually determined PD-L1 scores. However, manual scoring shows variability between different investigators and is influenced by cognitive and visual traps and could therefore negatively influence treatment decisions. Automated PD-L1 scoring could facilitate reliable and reproducible results. Our novel approach uses three neural networks sequentially applied for fully automated PD-L1 scoring of all three established PD-L1 scores: tumor proportion score (TPS), combined positive score (CPS) and tumor-infiltrating immune cell score (ICS). Our approach was validated using WSIs of HNSCC cases and compared with manual PD-L1 scoring by human investigators. The inter-rater correlation (ICC) between human and machine was very similar to the human-human correlation. The ICC was slightly higher between human-machine compared to human-human for the CPS and ICS, but a slightly lower for the TPS. Our study provides deeper insights into automated PD-L1 scoring by neural networks and its limitations. This may serve as a basis to improve ICI patient selection in the future.