Precision histology: how deep learning is poised to revitalize histomorphology for personalized cancer care. NPJ Precis Oncol. 2017;1:22. [PMID: 29872706 PMCID: PMC5871847 DOI: 10.1038/s41698-017-0022-1]

Radiomics at a Glance: A Few Lessons Learned from Learning Approaches

Simple Summary

Radiomics has become a prominent component of medical imaging research and many studies show its specific value as a support tool for clinical decision-making processes. Radiomic data are typically analyzed with statistical and machine learning methods, which change depending on the disease context and the imaging modality. We found a certain bias in the literature towards the use of such methods and believe that this limitation may influence the capacity of producing accurate and reliable decisions. Therefore, in view of the relevance of various types of learning methods, we report their significance and discuss their unrevealed potential.

Abstract

Processing and modeling medical images have traditionally represented complex tasks requiring multidisciplinary collaboration. The advent of radiomics has assigned a central role to quantitative data analytics targeting medical image features algorithmically extracted from large volumes of images. Apart from the ultimate goal of supporting diagnostic, prognostic, and therapeutic decisions, radiomics is computationally attractive due to specific strengths: scalability, efficiency, and precision. Optimization is achieved by highly sophisticated statistical and machine learning algorithms, but it is especially deep learning that stands out as the leading inference approach. Various types of hybrid learning can be considered when building complex integrative approaches aimed to deliver gains in accuracy for both classification and prediction tasks. This perspective reviews some selected learning methods by focusing on both their significance for radiomics and their unveiled potential.

Cellular resolution in clinical MALDI mass spectrometry imaging: the latest advancements and current challenges

Mass spectrometry (MS) is the workhorse of metabolomics, proteomics and lipidomics. Mass spectrometry imaging (MSI), its extension to spatially resolved analysis of tissues, is a powerful tool for visualizing molecular information within the histological context of tissue. This review summarizes recent developments in MSI and highlights current challenges that remain to achieve molecular imaging at the cellular level of clinical specimens. We focus on matrix-assisted laser desorption/ionization (MALDI)-MSI. We discuss the current status of each of the analysis steps and remaining challenges to reach the desired level of cellular imaging. Currently, analyte delocalization and degradation, matrix crystal size, laser focus restrictions and detector sensitivity are factors that are limiting spatial resolution. New sample preparation devices and laser optic systems are being developed to push the boundaries of these limitations. Furthermore, we review the processing of cellular MSI data and images, and the systematic integration of these data in the light of available algorithms and databases. We discuss roadblocks in the data analysis pipeline and show how technology from other fields can be used to overcome these. Finally, we conclude with curative and community efforts that are needed to enable contextualization of the information obtained.

The future of pathology is digital

Information, archives, and intelligent artificial systems are part of everyday life in modern medicine. They already support medical staff by mapping their workflows with shared availability of cases' referral information, as needed for example, by the pathologist, and this support will be increased in the future even more. In radiology, established standards define information models, data transmission mechanisms, and workflows. Other disciplines, such as pathology, cardiology, and radiation therapy, now define further demands in addition to these established standards. Pathology may have the highest technical demands on the systems, with very complex workflows, and the digitization of slides generating enormous amounts of data up to Gigabytes per biopsy. This requires enormous amounts of data to be generated per biopsy, up to the gigabyte range. Digital pathology allows a change from classical histopathological diagnosis with microscopes and glass slides to virtual microscopy on the computer, with multiple tools using artificial intelligence and machine learning to support pathologists in their future work.

Enhancing the Value of Histopathological Assessment of Allograft Biopsy Monitoring

Traditional histopathological allograft biopsy evaluation provides, within hours, diagnoses, prognostic information, and mechanistic insights into disease processes. However, proponents of an array of alternative monitoring platforms, broadly classified as "invasive" or "noninvasive" depending on whether allograft tissue is needed, question the value proposition of tissue histopathology. The authors explore the pros and cons of current analytical methods relative to the value of traditional and illustrate advancements of next-generation histopathological evaluation of tissue biopsies. We describe the continuing value of traditional histopathological tissue assessment and "next-generation pathology (NGP)," broadly defined as staining/labeling techniques coupled with digital imaging and automated image analysis. Noninvasive imaging and fluid (blood and urine) analyses promote low-risk, global organ assessment, and "molecular" data output, respectively; invasive alternatives promote objective, "mechanistic" insights by creating gene lists with variably increased/decreased expression compared with steady state/baseline. Proponents of alternative approaches contrast their preferred methods with traditional histopathology and: (1) fail to cite the main value of traditional and NGP-retention of spatial and inferred temporal context available for innumerable objective analyses and (2) belie an unfamiliarity with the impact of advances in imaging and software-guided analytics on emerging histopathology practices. Illustrative NGP examples demonstrate the value of multidimensional data that preserve tissue-based spatial and temporal contexts. We outline a path forward for clinical NGP implementation where "software-assisted sign-out" will enable pathologists to conduct objective analyses that can be incorporated into their final reports and improve patient care.

Spatially multiplexed RNA in situ hybridization to reveal tumor heterogeneity

Multiplexed RNA in situ hybridization for the analysis of gene expression patterns plays an important role in investigating development and disease. Here, we present a method for multiplexed RNA-ISH to detect spatial tumor heterogeneity in tissue sections. We made use of a microfluidic chip to deliver ISH-probes locally to regions of a few hundred micrometers over time periods of tens of minutes. This spatial multiplexing method can be combined with ISH-approaches based on signal amplification, with bright field detection and with the commonly used format of formalin-fixed paraffin-embedded tissue sections. By using this method, we analyzed the expression of HER2 with internal positive and negative controls (ActB, dapB) as well as predictive biomarker panels (ER, PgR, HER2) in a spatially multiplexed manner on single mammary carcinoma sections. We further demonstrated the applicability of the technique for subtype differentiation in breast cancer. Local analysis of HER2 revealed medium to high spatial heterogeneity of gene expression (Cohen effect size r = 0.4) in equivocally tested tumor tissues. Thereby, we exemplify the importance of using such a complementary approach for the analysis of spatial heterogeneity, in particular for equivocally tested tumor samples. As the method is compatible with a range of ISH approaches and tissue samples, it has the potential to find broad applicability in the context of molecular analysis of human diseases.

Three-dimensional modeling of human neurodegeneration: brain organoids coming of age

The prevalence of dementia and other neurodegenerative diseases is rapidly increasing in aging nations. These relentless and progressive diseases remain largely without disease-modifying treatments despite decades of research and investments. It is becoming clear that traditional two-dimensional culture and animal model systems, while providing valuable insights on the major pathophysiological pathways associated with these diseases, have not translated well to patients' bedside. Fortunately, the advent of induced-pluripotent stem cells and three-dimensional cell culture now provide tools that are revolutionizing the study of human diseases by permitting analysis of patient-derived human tissue with non-invasive procedures. Specifically, brain organoids, self-organizing neural structures that can mimic human fetal brain development, have now been harnessed to develop alternative models of Alzheimer's disease, Parkinson's disease, motor neuron disease, and Frontotemporal dementia by recapitulating important neuropathological hallmarks found in these disorders. Despite these early breakthroughs, several limitations need to be vetted in brain organoid models in order to more faithfully match human tissue qualities, including relative tissue immaturity, lack of vascularization and incomplete cellular diversity found in this culture system. Here, we review current brain organoid protocols, the pathophysiology of neurodegenerative disorders, and early studies with brain organoid neurodegeneration models. We then discuss the multiple engineering and conceptual challenges surrounding their use and provide possible solutions and exciting avenues to be pursued. Altogether, we believe that brain organoids models, improved with classical and emerging molecular and analytic tools, have the potential to unravel the opaque pathophysiological mechanisms of neurodegeneration and devise novel treatments for an array of neurodegenerative disorders.

Clinical implications of intratumor heterogeneity: challenges and opportunities

In this review, we highlight the role of intratumoral heterogeneity, focusing on the clinical and biological ramifications this phenomenon poses. Intratumoral heterogeneity arises through complex genetic, epigenetic, and protein modifications that drive phenotypic selection in response to environmental pressures. Functionally, heterogeneity provides tumors with significant adaptability. This ranges from mutual beneficial cooperation between cells, which nurture features such as growth and metastasis, to the narrow escape and survival of clonal cell populations that have adapted to thrive under specific conditions such as hypoxia or chemotherapy. These dynamic intercellular interplays are guided by a Darwinian selection landscape between clonal tumor cell populations and the tumor microenvironment. Understanding the involved drivers and functional consequences of such tumor heterogeneity is challenging but also promises to provide novel insight needed to confront the problem of therapeutic resistance in tumors.

A deep learning image-based intrinsic molecular subtype classifier of breast tumors reveals tumor heterogeneity that may affect survival

BACKGROUND

Breast cancer intrinsic molecular subtype (IMS) as classified by the expression-based PAM50 assay is considered a strong prognostic feature, even when controlled for by standard clinicopathological features such as age, grade, and nodal status, yet the molecular testing required to elucidate these subtypes is not routinely performed. Furthermore, when such bulk assays as RNA sequencing are performed, intratumoral heterogeneity that may affect prognosis and therapeutic decision-making can be missed.

METHODS

As a more facile and readily available method for determining IMS in breast cancer, we developed a deep learning approach for approximating PAM50 intrinsic subtyping using only whole-slide images of H&E-stained breast biopsy tissue sections. This algorithm was trained on images from 443 tumors that had previously undergone PAM50 subtyping to classify small patches of the images into four major molecular subtypes-Basal-like, HER2-enriched, Luminal A, and Luminal B-as well as Basal vs. non-Basal. The algorithm was subsequently used for subtype classification of a held-out set of 222 tumors.

RESULTS

This deep learning image-based classifier correctly subtyped the majority of samples in the held-out set of tumors. However, in many cases, significant heterogeneity was observed in assigned subtypes across patches from within a single whole-slide image. We performed further analysis of heterogeneity, focusing on contrasting Luminal A and Basal-like subtypes because classifications from our deep learning algorithm-similar to PAM50-are associated with significant differences in survival between these two subtypes. Patients with tumors classified as heterogeneous were found to have survival intermediate between Luminal A and Basal patients, as well as more varied levels of hormone receptor expression patterns.

CONCLUSIONS

Here, we present a method for minimizing manual work required to identify cancer-rich patches among all multiscale patches in H&E-stained WSIs that can be generalized to any indication. These results suggest that advanced deep machine learning methods that use only routinely collected whole-slide images can approximate RNA-seq-based molecular tests such as PAM50 and, importantly, may increase detection of heterogeneous tumors that may require more detailed subtype analysis.

Predicting primary site of secondary liver cancer with a neural estimator of metastatic origin

Purpose: Pathologists rely on relevant clinical information, visual inspection of stained tissue slide morphology, and sophisticated molecular diagnostics to accurately infer the biological origin of secondary metastatic cancer. While highly effective, this process is expensive in terms of time and clinical resources. We seek to develop and evaluate a computer vision system designed to reasonably infer metastatic origin of secondary liver cancer directly from digitized histopathological whole slide images of liver biopsy. Approach: We illustrate a two-stage deep learning approach to accomplish this task. We first train a model to identify spatially localized regions of cancerous tumor within digitized hematoxylin and eosin (H&E)-stained tissue sections of secondary liver cancer based on a pathologist's annotation of several whole slide images. Then, a second model is trained to generate predictions of the cancers' metastatic origin belonging to one of three distinct clinically relevant classes as confirmed by immunohistochemistry. Results: Our approach achieves a classification accuracy of 90.2% in determining metastatic origin of whole slide images from a held-out test set, which compares favorably to an established clinical benchmark by three board-certified pathologists whose accuracies ranged from 90.2% to 94.1% on the same prediction task. Conclusions: We illustrate the potential impact of deep learning systems to leverage morphological and structural features of H&E-stained tissue sections to guide pathological and clinical determination of the metastatic origin of secondary liver cancers.

Mass spectrometry imaging to detect lipid biomarkers and disease signatures in cancer

BACKGROUND

Current methods to identify, classify, and predict tumor behavior mostly rely on histology, immunohistochemistry, and molecular determinants. However, better predictive markers are required for tumor diagnosis and evaluation. Due, in part, to recent technological advancements, metabolomics and lipid biomarkers have become a promising area in cancer research. Therefore, there is a necessity for novel and complementary techniques to identify and visualize these molecular markers within tumors and surrounding tissue.

RECENT FINDINGS

Since its introduction, mass spectrometry imaging (MSI) has proven to be a powerful tool for mapping analytes in biological tissues. By adding the label-free specificity of mass spectrometry to the detailed spatial information of traditional histology, hundreds of lipids can be imaged simultaneously within a tumor. MSI provides highly detailed lipid maps for comparing intra-tumor, tumor margin, and healthy regions to identify biomarkers, patterns of disease, and potential therapeutic targets. In this manuscript, recent advancement in sample preparation and MSI technologies are discussed with special emphasis on cancer lipid research to identify tumor biomarkers.

CONCLUSION

MSI offers a unique approach for biomolecular characterization of tumor tissues and provides valuable complementary information to histology for lipid biomarker discovery and tumor classification in clinical and research cancer applications.

Automated detection and quantification of breast cancer brain metastases in an animal model using democratized machine learning tools

Advances in digital whole-slide imaging and machine learning (ML) provide new opportunities for automated examination and quantification of histopathological slides to support pathologists and biologists. However, implementation of ML tools often requires advanced skills in computer science that may not be immediately available in the traditional wet-lab environment. Here, we propose a simple and accessible workflow to automate detection and quantification of brain epithelial metastases on digitized histological slides. We leverage 100 Hematoxylin & Eosin (H&E)-stained whole slide images (WSIs) from 25 Balb/c mice with various level of brain metastatic tumor burden. A supervised training of the Trainable Weka Segmentation (TWS) from Fiji was achieved from annotated WSIs. Upon comparison with manually drawn regions, it is apparent that the algorithm learned to identify and segment cancer cell-specific nuclei and normal brain tissue. Our approach resulted in a robust and highly concordant correlation between automated metastases quantification of brain metastases and manual human assessment (R² = 0.8783; P < 0.0001). This simple approach is amenable to other similar analyses, including that of human tissues. Widespread adoption of these tools aims to democratize ML and improve precision in traditionally qualitative tasks in histopathology-based research.

System-based approaches as prognostic tools for glioblastoma

BACKGROUND

The evasion of apoptosis is a hallmark of cancer. Understanding this process holistically and overcoming apoptosis resistance is a goal of many research teams in order to develop better treatment options for cancer patients. Efforts are also ongoing to personalize the treatment of patients. Strategies to confirm the therapeutic efficacy of current treatments or indeed to identify potential novel additional options would be extremely beneficial to both clinicians and patients. In the past few years, system medicine approaches have been developed that model the biochemical pathways of apoptosis. These systems tools incorporate and analyse the complex biological networks involved. For their successful integration into clinical practice, it is mandatory to integrate systems approaches with routine clinical and histopathological practice to deliver personalized care for patients.

RESULTS

We review here the development of system medicine approaches that model apoptosis for the treatment of cancer with a specific emphasis on the aggressive brain cancer, glioblastoma.

CONCLUSIONS

We discuss the current understanding in the field and present new approaches that highlight the potential of system medicine approaches to influence how glioblastoma is diagnosed and treated in the future.

Society of Toxicologic Pathology Digital Pathology and Image Analysis Special Interest Group Article*: Opinion on the Application of Artificial Intelligence and Machine Learning to Digital Toxicologic Pathology

Toxicologic pathology is transitioning from analog to digital methods. This transition seems inevitable due to a host of ongoing social and medical technological forces. Of these, artificial intelligence (AI) and in particular machine learning (ML) are globally disruptive, rapidly growing sectors of technology whose impact on the long-established field of histopathology is quickly being realized. The development of increasing numbers of algorithms, peering ever deeper into the histopathological space, has demonstrated to the scientific community that AI pathology platforms are now poised to truly impact the future of precision and personalized medicine. However, as with all great technological advances, there are implementation and adoption challenges. This review aims to define common and relevant AI and ML terminology, describe data generation and interpretation, outline current and potential future business cases, discuss validation and regulatory hurdles, and most importantly, propose how overcoming the challenges of this burgeoning technology may shape toxicologic pathology for years to come, enabling pathologists to contribute even more effectively to answering scientific questions and solving global health issues. [Box: see text]

ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning

Machine learning algorithms hold the promise to effectively automate the analysis of histopathological images that are routinely generated in clinical practice. Any machine learning method used in the clinical diagnostic process has to be extremely accurate and, ideally, provide a measure of uncertainty for its predictions. Such accurate and reliable classifiers need enough labelled data for training, which requires time-consuming and costly manual annotation by pathologists. Thus, it is critical to minimise the amount of data needed to reach the desired accuracy by maximising the efficiency of training. We propose an accurate, reliable and active (ARA) image classification framework and introduce a new Bayesian Convolutional Neural Network (ARA-CNN) for classifying histopathological images of colorectal cancer. The model achieves exceptional classification accuracy, outperforming other models trained on the same dataset. The network outputs an uncertainty measurement for each tested image. We show that uncertainty measures can be used to detect mislabelled training samples and can be employed in an efficient active learning workflow. Using a variational dropout-based entropy measure of uncertainty in the workflow speeds up the learning process by roughly 45%. Finally, we utilise our model to segment whole-slide images of colorectal tissue and compute segmentation-based spatial statistics.

Connecting Histopathology Imaging and Proteomics in Kidney Cancer through Machine Learning

Proteomics data encode molecular features of diagnostic value and accurately reflect key underlying biological mechanisms in cancers. Histopathology imaging is a well-established clinical approach to cancer diagnosis. The predictive relationship between large-scale proteomics and H&E-stained histopathology images remains largely uncharacterized. Here we investigate such associations through the application of machine learning, including deep neural networks, to proteomics and histology imaging datasets generated by the Clinical Proteomic Tumor Analysis Consortium (CPTAC) from clear cell renal cell carcinoma patients. We report robust correlations between a set of diagnostic proteins and predictions generated by an imaging-based classification model. Proteins significantly correlated with the histology-based predictions are significantly implicated in immune responses, extracellular matrix reorganization, and metabolism. Moreover, we showed that the genes encoding these proteins also reliably recapitulate the biological associations with imaging-derived predictions based on strong gene–protein expression correlations. Our findings offer novel insights into the integrative modeling of histology and omics data through machine learning, as well as the methodological basis for new research opportunities in this and other cancer types.

Panoptic View of Prognostic Models for Personalized Breast Cancer Management

The efforts to personalize treatment for patients with breast cancer have led to a focus on the deeper characterization of genotypic and phenotypic heterogeneity among breast cancers. Traditional pathology utilizes microscopy to profile the morphologic features and organizational architecture of tumor tissue for predicting the course of disease, and is the first-line set of guiding tools for customizing treatment decision-making. Currently, clinicians use this information, combined with the disease stage, to predict patient prognosis to some extent. However, tumoral heterogeneity stubbornly persists among patient subgroups delineated by these clinicopathologic characteristics, as currently used methodologies in diagnostic pathology lack the capability to discern deeper genotypic and subtler phenotypic differences among individual patients. Recent advancements in molecular pathology, however, are poised to change this by joining forces with multiple-omics technologies (genomics, transcriptomics, epigenomics, proteomics, and metabolomics) that provide a wealth of data about the precise molecular complement of each patient's tumor. In addition, these technologies inform the drivers of disease aggressiveness, the determinants of therapeutic response, and new treatment targets in the individual patient. The tumor architecture information can be integrated with the knowledge of the detailed mutational, transcriptional, and proteomic phenotypes of cancer cells within individual tumors to derive a new level of biologic insight that enables powerful, data-driven patient stratification and customization of treatment for each patient, at each stage of the disease. This review summarizes the prognostic and predictive insights provided by commercially available gene expression-based tests and other multivariate or clinical -omics-based prognostic/predictive models currently under development, and proposes a more inclusive multiplatform approach to tackling the challenging heterogeneity of breast cancer to individualize its management. "The future is already here-it's just not very evenly distributed."-William Ford Gibson.

Molecular and histological correlations in liver cancer

Hepatocellular carcinoma (HCC) is a highly heterogeneous cancer, both at the molecular and histological level. High-throughput sequencing and gene expression profiling have identified distinct transcriptomic subclasses and numerous recurrent genetic alterations; several HCC subtypes characterised by histological features have also been identified. HCC phenotype appears to be closely related to particular gene mutations, tumour subgroups and/or oncogenic pathways. Non-proliferative tumours display a well-differentiated phenotype. Among this molecular subgroup, CTNNB1-mutated HCCs constitute a homogeneous subtype, exhibiting cholestasis and microtrabecular and pseudoglandular architectural patterns. Another non-proliferative subtype has a gene expression pattern similar to that of mature hepatocytes (G4) and displays a steatohepatitic phenotype. In contrast, proliferative HCCs are most often poorly differentiated, and notably include tumours with progenitor features. A novel morphological variant of proliferative HCC - designated "macrotrabecular-massive" - was recently shown to be associated with angiogenesis activation and poor prognosis. Altogether, these findings may help to translate our knowledge of HCC biology into clinical practice, resulting in improved precision medicine for patients with this highly aggressive malignancy. This manuscript reviews the most recent data in this exciting field, discussing future directions and challenges.

Pan-Renal Cell Carcinoma classification and survival prediction from histopathology images using deep learning

Histopathological images contain morphological markers of disease progression that have diagnostic and predictive values. In this study, we demonstrate how deep learning framework can be used for an automatic classification of Renal Cell Carcinoma (RCC) subtypes, and for identification of features that predict survival outcome from digital histopathological images. Convolutional neural networks (CNN's) trained on whole-slide images distinguish clear cell and chromophobe RCC from normal tissue with a classification accuracy of 93.39% and 87.34%, respectively. Further, a CNN trained to distinguish clear cell, chromophobe and papillary RCC achieves a classification accuracy of 94.07%. Here, we introduced a novel support vector machine-based method that helped to break the multi-class classification task into multiple binary classification tasks which not only improved the performance of the model but also helped to deal with data imbalance. Finally, we extracted the morphological features from high probability tumor regions identified by the CNN to predict patient survival outcome of most common clear cell RCC. The generated risk index based on both tumor shape and nuclei features are significantly associated with patient survival outcome. These results highlight that deep learning can play a role in both cancer diagnosis and prognosis.

Artificial Intelligence Algorithms to Assess Hormonal Status From Tissue Microarrays in Patients With Breast Cancer

IMPORTANCE

Immunohistochemistry (IHC) is the most widely used assay for identification of molecular biomarkers. However, IHC is time consuming and costly, depends on tissue-handling protocols, and relies on pathologists' subjective interpretation. Image analysis by machine learning is gaining ground for various applications in pathology but has not been proposed to replace chemical-based assays for molecular detection.

OBJECTIVE

To assess the prediction feasibility of molecular expression of biomarkers in cancer tissues, relying only on tissue architecture as seen in digitized hematoxylin-eosin (H&E)-stained specimens.

DESIGN, SETTING, AND PARTICIPANTS

This single-institution retrospective diagnostic study assessed the breast cancer tissue microarrays library of patients from Vancouver General Hospital, British Columbia, Canada. The study and analysis were conducted from July 1, 2015, through July 1, 2018. A machine learning method, termed morphological-based molecular profiling (MBMP), was developed. Logistic regression was used to explore correlations between histomorphology and biomarker expression, and a deep convolutional neural network was used to predict the biomarker expression in examined tissues.

MAIN OUTCOMES AND MEASURES

Positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristics curve measures of MBMP for assessment of molecular biomarkers.

RESULTS

The database consisted of 20 600 digitized, publicly available H&E-stained sections of 5356 patients with breast cancer from 2 cohorts. The median age at diagnosis was 61 years for cohort 1 (412 patients) and 62 years for cohort 2 (4944 patients), and the median follow-up was 12.0 years and 12.4 years, respectively. Tissue histomorphology was significantly correlated with the molecular expression of all 19 biomarkers assayed, including estrogen receptor (ER), progesterone receptor (PR), and ERBB2 (formerly HER2). Expression of ER was predicted for 105 of 207 validation patients in cohort 1 (50.7%) and 1059 of 2046 validation patients in cohort 2 (51.8%), with PPVs of 97% and 98%, respectively, NPVs of 68% and 76%, respectively, and accuracy of 91% and 92%, respectively, which were noninferior to traditional IHC (PPV, 91%-98%; NPV, 51%-78%; and accuracy, 81%-90%). Diagnostic accuracy improved given more data. Morphological analysis of patients with ER-negative/PR-positive status by IHC revealed resemblance to patients with ER-positive status (Bhattacharyya distance, 0.03) and not those with ER-negative/PR-negative status (Bhattacharyya distance, 0.25). This suggests a false-negative IHC finding and warrants antihormonal therapy for these patients.

CONCLUSIONS AND RELEVANCE

For at least half of the patients in this study, MBMP appeared to predict biomarker expression with noninferiority to IHC. Results suggest that prediction accuracy is likely to improve as data used for training expand. Morphological-based molecular profiling could be used as a general approach for mass-scale molecular profiling based on digitized H&E-stained images, allowing quick, accurate, and inexpensive methods for simultaneous profiling of multiple biomarkers in cancer tissues.

Combining Diagnostic Imaging and Pathology for Improving Diagnosis and Prognosis of Cancer

In the era of personalized medicine, the management of oncological patients requires a translational and multidisciplinary approach. During early phases of cancer development, biochemical alterations of cell metabolism occur much before the formation of detectable tumour masses. Current molecular imaging techniques, targeted to the study of molecular kinetics, employ molecular tracers capable of detecting cancer lesions with both high sensitivity and specificity while also providing essential information for both prognosis and therapy. On the contrary, complementary and crucial information is provided by histopathological examination and ancillary techniques such as immunohistochemistry. Thus, the successful collaboration between diagnostic imaging and anatomic pathology can represent a fundamental step in the "tortuous" but decisive path towards personalized medicine.

Artificial intelligence in cytopathology: a review of the literature and overview of commercial landscape

Artificial intelligence (AI) has made impressive strides recently in interpreting complex images, thanks to improvements in deep learning techniques and increasing computational power. Researchers have started applying these advanced techniques to pathology images, although most efforts have been focused on histopathology. Cytopathology, however, remains the original field of pathology for which AI models for clinical use were successfully commercialized, to assist with automating Papanicolaou test screening. Recent AI efforts have focused on whole slide images of both gynecologic and non-gynecologic cytopathology. This review summarizes the literature and commercial landscape of AI as applied to cytopathology.

Towards personalized computational oncology: from spatial models of tumour spheroids, to organoids, to tissues

A main goal of mathematical and computational oncology is to develop quantitative tools to determine the most effective therapies for each individual patient. This involves predicting the right drug to be administered at the right time and at the right dose. Such an approach is known as precision medicine. Mathematical modelling can play an invaluable role in the development of such therapeutic strategies, since it allows for relatively fast, efficient and inexpensive simulations of a large number of treatment schedules in order to find the most effective. This review is a survey of mathematical models that explicitly take into account the spatial architecture of three-dimensional tumours and address tumour development, progression and response to treatments. In particular, we discuss models of epithelial acini, multicellular spheroids, normal and tumour spheroids and organoids, and multi-component tissues. Our intent is to showcase how these in silico models can be applied to patient-specific data to assess which therapeutic strategies will be the most efficient. We also present the concept of virtual clinical trials that integrate standard-of-care patient data, medical imaging, organ-on-chip experiments and computational models to determine personalized medical treatment strategies.

Radiomics in Glioblastoma: Current Status and Challenges Facing Clinical Implementation

Radiomics analysis has had remarkable progress along with advances in medical imaging, most notability in central nervous system malignancies. Radiomics refers to the extraction of a large number of quantitative features that describe the intensity, texture and geometrical characteristics attributed to the tumor radiographic data. These features have been used to build predictive models for diagnosis, prognosis, and therapeutic response. Such models are being combined with clinical, biological, genetics and proteomic features to enhance reproducibility. Broadly, the four steps necessary for radiomic analysis are: (1) image acquisition, (2) segmentation or labeling, (3) feature extraction, and (4) statistical analysis. Major methodological challenges remain prior to clinical implementation. Essential steps include: adoption of an optimized standard imaging process, establishing a common criterion for performing segmentation, fully automated extraction of radiomic features without redundancy, and robust statistical modeling validated in the prospective setting. This review walks through these steps in detail, as it pertains to high grade gliomas. The impact on precision medicine will be discussed, as well as the challenges facing clinical implementation of radiomic in the current management of glioblastoma.

Impact of pre-analytical variables on deep learning accuracy in histopathology

AIMS

Machine learning (ML) binary classification in diagnostic histopathology is an area of intense investigation. Several assumptions, including training image quality/format and the number of training images required, appear to be similar in many studies irrespective of the paucity of supporting evidence. We empirically compared training image file type, training set size, and two common convolutional neural networks (CNNs) using transfer learning (ResNet50 and SqueezeNet).

METHODS AND RESULTS

Thirty haematoxylin and eosin (H&E)-stained slides with carcinoma or normal tissue from three tissue types (breast, colon, and prostate) were photographed, generating 3000 partially overlapping images (1000 per tissue type). These lossless Portable Networks Graphics (PNGs) images were converted to lossy Joint Photographic Experts Group (JPG) images. Tissue type-specific binary classification ML models were developed by the use of all PNG or JPG images, and repeated with a subset of 500, 200, 100, 50, 30 and 10 images. Eleven models were generated for each tissue type, at each quantity of training images, for each file type, and for each CNN, resulting in 924 models. Internal accuracies and generalisation accuracies were compared. There was no meaningful significant difference in accuracies between PNG and JPG models. Models trained with more images did not invariably perform better. ResNet50 typically outperformed SqueezeNet. Models were generalisable within a tissue type but not across tissue types.

CONCLUSIONS

Lossy JPG images were not inferior to lossless PNG images in our models. Large numbers of unique H&E-stained slides were not required for training optimal ML models. This reinforces the need for an evidence-based approach to best practices for histopathological ML.

Physician perspectives on integration of artificial intelligence into diagnostic pathology

Advancements in computer vision and artificial intelligence (AI) carry the potential to make significant contributions to health care, particularly in diagnostic specialties such as radiology and pathology. The impact of these technologies on physician stakeholders is the subject of significant speculation. There is however a dearth of information regarding the opinions, enthusiasm, and concerns of the pathology community at large. Here, we report results from a survey of 487 pathologist-respondents practicing in 54 countries, conducted to examine perspectives on AI implementation in clinical practice. Despite limitations, including difficulty with quantifying response bias and verifying identity of respondents to this anonymous and voluntary survey, several interesting findings were uncovered. Overall, respondents carried generally positive attitudes towards AI, with nearly 75% reporting interest or excitement in AI as a diagnostic tool to facilitate improvements in workflow efficiency and quality assurance in pathology. Importantly, even within the more optimistic cohort, a significant number of respondents endorsed concerns about AI, including the potential for job displacement and replacement. Overall, around 80% of respondents predicted the introduction of AI technology in the pathology laboratory within the coming decade. Attempts to identify statistically significant demographic characteristics (e.g., age, sex, type/place of practice) predictive of attitudes towards AI using Kolmogorov–Smirnov (KS) testing revealed several associations. Important themes which were commented on by respondents included the need for increasing efforts towards physician training and resolving medical-legal implications prior to the generalized implementation of AI in pathology.

Precise definition of PTEN C-terminal epitopes and its implications in clinical oncology

Anti-PTEN monoclonal antibodies (mAb) are arising as important tools for immunohistochemistry (IHC) and protein quantification routine analysis in clinical oncology. Although an effort has been made to document the reliability of tumor tissue section immunostaining by anti-PTEN mAb, and to standardize their IHC use in research and in the clinical practice, the precise topological and biochemical definition of the epitope recognized by each mAb has been conventionally overlooked. In this study, six commercial anti-PTEN mAb have been validated and characterized for sensitivity and specificity by IHC and FISH, using a set of prostate and urothelial bladder tumor specimens, and by immunoblot, using PTEN positive and PTEN negative human cell lines. Immunoblot precise epitope mapping, performed using recombinant PTEN variants and mutations, revealed that all mAb recognized linear epitopes of 6–11 amino acid length at the PTEN C-terminus. Tumor-associated or disease-associated mutations at the PTEN C-terminus did not affect subcellular localization or PIP3 phosphatase activity of PTEN in cells, although resulted in specific loss of reactivity for some mAb. Furthermore, specific mimicking-phosphorylation mutations at the PTEN C-terminal region also abolished binding of specific mAb. Our study adds new evidence on the relevance of a precise epitope mapping in the validation of anti-PTEN mAb for their use in the clinics. This will be substantial to provide a more accurate diagnosis in clinical oncology based on PTEN protein expression in tumors and biological fluids.

Rapid histology of laryngeal squamous cell carcinoma with deep-learning based stimulated Raman scattering microscopy

Maximal resection of tumor while preserving the adjacent healthy tissue is particularly important for larynx surgery, hence precise and rapid intraoperative histology of laryngeal tissue is crucial for providing optimal surgical outcomes. We hypothesized that deep-learning based stimulated Raman scattering (SRS) microscopy could provide automated and accurate diagnosis of laryngeal squamous cell carcinoma on fresh, unprocessed surgical specimens without fixation, sectioning or staining. Methods: We first compared 80 pairs of adjacent frozen sections imaged with SRS and standard hematoxylin and eosin histology to evaluate their concordance. We then applied SRS imaging on fresh surgical tissues from 45 patients to reveal key diagnostic features, based on which we have constructed a deep learning based model to generate automated histologic results. 18,750 SRS fields of views were used to train and cross-validate our 34-layered residual convolutional neural network, which was used to classify 33 untrained fresh larynx surgical samples into normal and neoplasia. Furthermore, we simulated intraoperative evaluation of resection margins on totally removed larynxes. Results: We demonstrated near-perfect diagnostic concordance (Cohen's kappa, κ > 0.90) between SRS and standard histology as evaluated by three pathologists. And deep-learning based SRS correctly classified 33 independent surgical specimens with 100% accuracy. We also demonstrated that our method could identify tissue neoplasia at the simulated resection margins that appear grossly normal with naked eyes. Conclusion: Our results indicated that SRS histology integrated with deep learning algorithm provides potential for delivering rapid intraoperative diagnosis that could aid the surgical management of laryngeal cancer.

Construction and Utilization of a Neural Network Model to Predict Current Procedural Terminology Codes from Pathology Report Texts

Background:

At our department, each specimen was assigned a tentative current procedural terminology (CPT) code at accessioning. The codes were subject to subsequent changes by pathologist assistants and pathologists. After the cases had been finalized, their CPT codes went through a final verification step by coding staff, with the aid of a keyword-based CPT code-checking web application. Greater than 97% of the initial assignments were correct. This article describes the construction of a CPT code-predicting neural network model and its incorporation into the CPT code-checking application.

Materials and Methods:

R programming language was used. Pathology report texts and CPT codes for the cases finalized during January 1–November 30, 2018, were retrieved from the database. The order of the specimens was randomized before the data were partitioned into training and validation set. R Keras package was used for both model training and prediction. The chosen neural network had a three-layer architecture consisting of a word-embedding layer, a bidirectional long short-term memory (LSTM) layer, and a densely connected layer. It used concatenated header-diagnosis texts as the input.

Results:

The model predicted CPT codes in both the validation data set and the test data set with an accuracy of 97.5% and 97.6%, respectively. Closer examination of the test data set (cases from December 1 to 27, 2018) revealed two interesting observations. First, among the specimens that had incorrect initial CPT code assignments, the model disagreed with the initial assignments in 73.6% (117/159) and agreed in 26.4% (42/159). Second, the model identified nine additional specimens with incorrect CPT codes that had evaded all steps of checking.

Conclusions:

A neural network model using report texts to predict CPT codes can achieve high accuracy in prediction and moderate sensitivity in error detection. Neural networks may play increasing roles in CPT coding in surgical pathology.

Precision immunoprofiling by image analysis and artificial intelligence

Clinical success of immunotherapy is driving the need for new prognostic and predictive assays to inform patient selection and stratification. This requirement can be met by a combination of computational pathology and artificial intelligence. Here, we critically assess computational approaches supporting the development of a standardized methodology in the assessment of immune-oncology biomarkers, such as PD-L1 and immune cell infiltrates. We examine immunoprofiling through spatial analysis of tumor-immune cell interactions and multiplexing technologies as a predictor of patient response to cancer treatment. Further, we discuss how integrated bioinformatics can enable the amalgamation of complex morphological phenotypes with the multiomics datasets that drive precision medicine. We provide an outline to machine learning (ML) and artificial intelligence tools and illustrate fields of application in immune-oncology, such as pattern-recognition in large and complex datasets and deep learning approaches for survival analysis. Synergies of surgical pathology and computational analyses are expected to improve patient stratification in immuno-oncology. We propose that future clinical demands will be best met by (1) dedicated research at the interface of pathology and bioinformatics, supported by professional societies, and (2) the integration of data sciences and digital image analysis in the professional education of pathologists.

Discriminating normal regions within cancerous hen ovarian tissue using multivariate hyperspectral image analysis

RATIONALE

Identification of subregions under different pathological conditions on cancerous tissue is of great significance for understanding cancer progression and metastasis. Infrared matrix-assisted laser desorption electrospray ionization mass spectrometry (IR-MALDESI-MS) can be potentially used for diagnostic purposes since it can monitor spatial distribution and abundance of metabolites and lipids in biological tissues. However, the large size and high dimensionality of hyperspectral data make analysis and interpretation challenging. To overcome these barriers, multivariate methods were applied to IR-MALDESI data for the first time, aiming at efficiently resolving mass spectral images, from which these results were then used to identify normal regions within cancerous tissue.

METHODS

Molecular profiles of healthy and cancerous hen ovary tissues were generated by IR-MALDESI-MS. Principal component analysis (PCA) combined with color-coding built a single tissue image which summarizes the high-dimensional data features. Pixels with similar color indicated similar composition. PCA results from healthy tissue were further used to test each pixel in cancerous tissue to determine if it is healthy. Multivariate curve resolution-alternating least squares (MCR-ALS) was used to obtain major spatial features existing in ovary tissues, and group molecules with the same distribution patterns simultaneously.

RESULTS

PCA as the predominating dimensionality reduction approach captured over 90% spectral variances by the first three PCs. The PCA images show the cancerous tissue is more chemically heterogeneous than healthy tissue, where at least four regions with different m/z profiles can be differentiated. PCA modeling assigns top regions of cancerous tissue as healthy-like. MCR-ALS extracted three and four major compounds from healthy and cancerous tissue, respectively. Evaluating similarities of resolved spectra uncovered the chemical components that were distinct in some regions on cancerous tissue, serving as a supplementary way to differentiate healthy and cancerous regions.

CONCLUSIONS

Two unsupervised chemometric methods including PCA and MCR-ALS were applied for resolving and visualizing IR-MALDESI-MS data acquired from hen ovary tissues, improving the interpretation of mass spectrometry imaging results. Then possible normal regions were differentiated from cancerous tissue sections. No prior knowledge is required using either chemometric method, so our approach is readily suitable for unstained tissue samples, which allows one to reveal the molecular events happening during disease progression.

Deep learning for image analysis: Personalizing medicine closer to the point of care

The precision-based revolution in medicine continues to demand stratification of patients into smaller and more personalized subgroups. While genomic technologies have largely led this movement, diagnostic results can take days to weeks to generate. Management at, or closer to, the point of care still heavily relies on the subjective qualitative interpretation of clinical and diagnostic imaging findings. New and emerging technological advances in artificial intelligence (AI) now appear poised to help bring objectivity and precision to these traditionally qualitative analytic tools. In particular, one specific form of AI, known as deep learning, is achieving expert-level disease classifications in many areas of diagnostic medicine dependent on visual and image-based findings. Here, we briefly review concepts of deep learning, and more specifically recent developments in convolutional neural networks (CNNs), to highlight their transformative potential in personalized medicine and, in particular, diagnostic histopathology. Understanding the opportunities and challenges of these quantitative machine-based decision support tools is critical to their widespread introduction into routine diagnostics.

Digital pathology: semper ad meliora

This review is an evidence-based summary of digital pathology: past, present and future. It discusses digital surgical pathology and the cytopathology digitisation challenge as well as the performance of digital histopathology and cytopathology as a diagnostic tool, particularly in contrast to user perceptions. Time and cost efficiency of digital pathology, learning curves, education and quality assurance, with the importance of validation of systems, is emphasised. The review concludes with a discussion of digital pathology as a source of 'big data' and where this might lead pathologists in the digital pathology future.

Deep Learning Global Glomerulosclerosis in Transplant Kidney Frozen Sections

Transplantable kidneys are in very limited supply. Accurate viability assessment prior to transplantation could minimize organ discard. Rapid and accurate evaluation of intra-operative donor kidney biopsies is essential for determining which kidneys are eligible for transplantation. The criterion for accepting or rejecting donor kidneys relies heavily on pathologist determination of the percent of glomeruli (determined from a frozen section) that are normal and sclerotic. This percentage is a critical measurement that correlates with transplant outcome. Inter- and intra-observer variability in donor biopsy evaluation is, however, significant. An automated method for determination of percent global glomerulosclerosis could prove useful in decreasing evaluation variability, increasing throughput, and easing the burden on pathologists. Here, we describe the development of a deep learning model that identifies and classifies non-sclerosed and sclerosed glomeruli in whole-slide images of donor kidney frozen section biopsies. This model extends a convolutional neural network (CNN) pre-trained on a large database of digital images. The extended model, when trained on just 48 whole slide images, exhibits slide-level evaluation performance on par with expert renal pathologists. Encouragingly, the model's performance is robust to slide preparation artifacts associated with frozen section preparation. The model substantially outperforms a model trained on image patches of isolated glomeruli, in terms of both accuracy and speed. The methodology overcomes the technical challenge of applying a pretrained CNN bottleneck model to whole-slide image classification. The traditional patch-based approach, while exhibiting deceptively good performance classifying isolated patches, does not translate successfully to whole-slide image segmentation in this setting. As the first model reported that identifies and classifies normal and sclerotic glomeruli in frozen kidney sections, and thus the first model reported in the literature relevant to kidney transplantation, it may become an essential part of donor kidney biopsy evaluation in the clinical setting.

Prospects and challenges for clinical decision support in the era of big data

Recently, there has been burgeoning interest in developing more effective and robust clinical decision support systems (CDSSs) for oncology. This has been primarily driven by the demands for more personalized and precise medical practice in oncology in the era of so-called Big Data (BD); an era that promises to harness the power of large-scale data flow to revolutionize cancer treatment. This interest in BD analytics has created new opportunities as well as new unmet challenges. These include: routine aggregation and standardization of clinical data; patient privacy; transformation of current analytical approaches to handle such noisy and heterogeneous data; and expanded use of advanced statistical learning methods based on confluence of modern statistical methods and machine learning algorithms. In this review, we present the current status of CDSSs in oncology, the prospects and current challenges of BD analytics, and the promising role of integrated modern statistics and machine learning algorithms in predicting complex clinical endpoints, individualizing treatment rules, and optimizing dynamic personalized treatment regimens. We discuss issues pertaining to these topics and present application examples from an aggregate of experiences. We also discuss the role of human factors in improving the utilization and acceptance of such enhanced CDSSs and how to mitigate possible sources of human error to achieve optimal performance and wider acceptance.

Hexa-functional tumour-seeking nano voyagers and annihilators for synergistic cancer theranostic applications

In order to meet the unmet medical needs for effective cancer treatment, multifunctional nanocarriers based on iron oxide nanoparticles hold tremendous promise. Here we report a superparamagnetic iron oxide nanoparticles based hexa-functional nanosystem for synergistic cancer theranostic applications by offering active tumour targeting, accumulation and complementary imaging capability by combining magnetic resonance imaging as well as near-infrared fluorescence, magnetophotothermia and chemotherapy. The uniquely designed nanosystem exhibited a paramount increase in the antitumour efficacy through the simultaneous application of multiple thermal effects called magnetophotothermia, which outweighed the therapeutic efficacy of the current thermo-chemo therapies or stand-alone therapies. The active tumour-seeking property with prolonged tumour accumulation and complementary imaging capability with improved sensitivity and resolution also augments the therapeutic efficacy of the proposed nanosystem. Additionally, the work proposes a deep-learning-based tumour cell nuclei detection technique from H&E stained images in anticipation of providing much inspiration for the future of precision histology.

Advances in the computational and molecular understanding of the prostate cancer cell nucleus

Nuclear alterations are a hallmark of many types of cancers, including prostate cancer (PCa). Recent evidence shows that subvisual changes, ones that may not be visually perceptible to a pathologist, to the nucleus and its ultrastructural components can precede visual histopathological recognition of cancer. Alterations to nuclear features, such as nuclear size and shape, texture, and spatial architecture, reflect the complex molecular-level changes that occur during oncogenesis. Quantitative nuclear morphometry, a field that uses computational approaches to identify and quantify malignancy-induced nuclear changes, can enable a detailed and objective analysis of the PCa cell nucleus. Recent advances in machine learning-based approaches can now automatically mine data related to these changes to aid in the diagnosis, decision making, and prediction of PCa prognoses. In this review, we use PCa as a case study to connect the molecular-level mechanisms that underlie these nuclear changes to the machine learning computational approaches, bridging the gap between the clinical and computational understanding of PCa. First, we will discuss recent developments to our understanding of the molecular events that drive nuclear alterations in the context of PCa: the role of the nuclear matrix and lamina in size and shape changes, the role of 3-dimensional chromatin organization and epigenetic modifications in textural changes, and the role of the tumor microenvironment in altering nuclear spatial topology. We will then discuss the advances in the applications of machine learning algorithms to automatically segment nuclei in prostate histopathological images, extract nuclear features to aid in diagnostic decision making, and predict potential outcomes, such as biochemical recurrence and survival. Finally, we will discuss the challenges and opportunities associated with translation of the quantitative nuclear morphometry methodology into the clinical space. Ultimately, accurate identification and quantification of nuclear alterations can contribute to the field of nucleomics and has applications for computationally driven precision oncologic patient care.

From hype to reality: data science enabling personalized medicine

BACKGROUND

Personalized, precision, P4, or stratified medicine is understood as a medical approach in which patients are stratified based on their disease subtype, risk, prognosis, or treatment response using specialized diagnostic tests. The key idea is to base medical decisions on individual patient characteristics, including molecular and behavioral biomarkers, rather than on population averages. Personalized medicine is deeply connected to and dependent on data science, specifically machine learning (often named Artificial Intelligence in the mainstream media). While during recent years there has been a lot of enthusiasm about the potential of 'big data' and machine learning-based solutions, there exist only few examples that impact current clinical practice. The lack of impact on clinical practice can largely be attributed to insufficient performance of predictive models, difficulties to interpret complex model predictions, and lack of validation via prospective clinical trials that demonstrate a clear benefit compared to the standard of care. In this paper, we review the potential of state-of-the-art data science approaches for personalized medicine, discuss open challenges, and highlight directions that may help to overcome them in the future.

CONCLUSIONS

There is a need for an interdisciplinary effort, including data scientists, physicians, patient advocates, regulatory agencies, and health insurance organizations. Partially unrealistic expectations and concerns about data science-based solutions need to be better managed. In parallel, computational methods must advance more to provide direct benefit to clinical practice.

Validation of the systematic scoring of immunohistochemically stained tumour tissue microarrays using QuPath digital image analysis

AIMS

Output from biomarker studies involving immunohistochemistry applied to tissue microarrays (TMA) is limited by the lack of an efficient and reproducible scoring methodology. In this study, we examine the functionality and reproducibility of biomarker scoring using the new, open-source, digital image analysis software, QuPath.

METHODS AND RESULTS

Three different reviewers, with varying experience of digital pathology and image analysis, applied an agreed QuPath scoring methodology to CD3 and p53 immunohistochemically stained TMAs from a colon cancer cohort (n = 661). Manual assessment was conducted by one reviewer for CD3. Survival analyses were conducted and intra- and interobserver reproducibility assessed. Median raw scores differed significantly between reviewers, but this had little impact on subsequent analyses. Lower CD3 scores were detected in cases who died from colorectal cancer compared to control cases, and this finding was significant for all three reviewers (P-value range = 0.002-0.02). Higher median p53 scores were generated among cases who died from colorectal cancer compared with controls (P-value range = 0.04-0.12). The ability to dichomotise cases into high versus low expression of CD3 and p53 showed excellent agreement between all three reviewers (kappa score range = 0.82-0.93). All three reviewers produced dichotomised expression scores that resulted in very similar hazard ratios for colorectal cancer-specific survival for each biomarker. Results from manual and QuPath methods of CD3 scoring were comparable, but QuPath scoring revealed stronger prognostic stratification.

CONCLUSIONS

Scoring of immunohistochemically stained tumour TMAs using QuPath is functional and reproducible, even among users of limited experience of digital pathology images, and more accurate than manual scoring.

Artificial intelligence in radiology

Artificial intelligence (AI) algorithms, particularly deep learning, have demonstrated remarkable progress in image-recognition tasks. Methods ranging from convolutional neural networks to variational autoencoders have found myriad applications in the medical image analysis field, propelling it forward at a rapid pace. Historically, in radiology practice, trained physicians visually assessed medical images for the detection, characterization and monitoring of diseases. AI methods excel at automatically recognizing complex patterns in imaging data and providing quantitative, rather than qualitative, assessments of radiographic characteristics. In this Opinion article, we establish a general understanding of AI methods, particularly those pertaining to image-based tasks. We explore how these methods could impact multiple facets of radiology, with a general focus on applications in oncology, and demonstrate ways in which these methods are advancing the field. Finally, we discuss the challenges facing clinical implementation and provide our perspective on how the domain could be advanced.

Artificial intelligence in radiology

Artificial intelligence (AI) algorithms, particularly deep learning, have demonstrated remarkable progress in image-recognition tasks. Methods ranging from convolutional neural networks to variational autoencoders have found myriad applications in the medical image analysis field, propelling it forward at a rapid pace. Historically, in radiology practice, trained physicians visually assessed medical images for the detection, characterization and monitoring of diseases. AI methods excel at automatically recognizing complex patterns in imaging data and providing quantitative, rather than qualitative, assessments of radiographic characteristics. In this Opinion article, we establish a general understanding of AI methods, particularly those pertaining to image-based tasks. We explore how these methods could impact multiple facets of radiology, with a general focus on applications in oncology, and demonstrate ways in which these methods are advancing the field. Finally, we discuss the challenges facing clinical implementation and provide our perspective on how the domain could be advanced.

Prospects for Theranostics in Neurosurgical Imaging: Empowering Confocal Laser Endomicroscopy Diagnostics via Deep Learning

Confocal laser endomicroscopy (CLE) is an advanced optical fluorescence imaging technology that has potential to increase intraoperative precision, extend resection, and tailor surgery for malignant invasive brain tumors because of its subcellular dimension resolution. Despite its promising diagnostic potential, interpreting the gray tone fluorescence images can be difficult for untrained users. CLE images can be distorted by motion artifacts, fluorescence signals out of detector dynamic range, or may be obscured by red blood cells, and thus interpreted as nondiagnostic (ND). However, just a single CLE image with a detectable pathognomonic histological tissue signature can suffice for intraoperative diagnosis. Dealing with the abundance of images from CLE is not unlike sifting through a myriad of genes, proteins, or other structural or metabolic markers to find something of commonality or uniqueness in cancer that might indicate a potential treatment scheme or target. In this review, we provide a detailed description of bioinformatical analysis methodology of CLE images that begins to assist the neurosurgeon and pathologist to rapidly connect on-the-fly intraoperative imaging, pathology, and surgical observation into a conclusionary system within the concept of theranostics. We present an overview and discuss deep learning models for automatic detection of the diagnostic CLE images and discuss various training regimes and ensemble modeling effect on power of deep learning predictive models. Two major approaches reviewed in this paper include the models that can automatically classify CLE images into diagnostic/ND, glioma/nonglioma, tumor/injury/normal categories, and models that can localize histological features on the CLE images using weakly supervised methods. We also briefly review advances in the deep learning approaches used for CLE image analysis in other organs. Significant advances in speed and precision of automated diagnostic frame selection would augment the diagnostic potential of CLE, improve operative workflow, and integration into brain tumor surgery. Such technology and bioinformatics analytics lend themselves to improved precision, personalization, and theranostics in brain tumor treatment.

Adversarial Time-to-Event Modeling

Modern health data science applications leverage abundant molecular and electronic health data, providing opportunities for machine learning to build statistical models to support clinical practice. Time-to-event analysis, also called survival analysis, stands as one of the most representative examples of such statistical models. We present a deep-network-based approach that leverages adversarial learning to address a key challenge in modern time-to-event modeling: nonparametric estimation of event-time distributions. We also introduce a principled cost function to exploit information from censored events (events that occur subsequent to the observation window). Unlike most time-to-event models, we focus on the estimation of time-to-event distributions, rather than time ordering. We validate our model on both benchmark and real datasets, demonstrating that the proposed formulation yields significant performance gains relative to a parametric alternative, which we also propose.