An Intelligent Search & Retrieval System (IRIS) and Clinical and Research Repository for Decision Support Based on Machine Learning and Joint Kernel-based Supervised Hashing

Large-scale, multi-site collaboration is becoming indispensable for a wide range of research and clinical activities in oncology. To facilitate the next generation of advances in cancer biology, precision oncology and the population sciences it will be necessary to develop and implement data management and analytic tools that empower investigators to reliably and objectively detect, characterize and chronicle the phenotypic and genomic changes that occur during the transformation from the benign to cancerous state and throughout the course of disease progression. To facilitate these efforts it is incumbent upon the informatics community to establish the workflows and architectures that automate the aggregation and organization of a growing range and number of clinical data types and modalities ranging from new molecular and laboratory tests to sophisticated diagnostic imaging studies. In an attempt to meet those challenges, leading health care centers across the country are making steep investments to establish enterprise-wide, data warehouses. A significant limitation of many data warehouses, however, is that they are designed to support only alphanumeric information. In contrast to those traditional designs, the system that we have developed supports automated collection and mining of multimodal data including genomics, digital pathology and radiology images. In this paper, our team describes the design, development and implementation of a multi-modal, Clinical & Research Data Warehouse (CRDW) that is tightly integrated with a suite of computational and machine-learning tools to provide actionable insight into the underlying characteristics of the tumor environment that would not be revealed using standard methods and tools. The System features a flexible Extract, Transform and Load (ETL) interface that enables it to adapt to aggregate data originating from different clinical and research sources depending on the specific EHR and other data sources utilized at a given deployment site.

An Expandable Informatics Framework for Enhancing Central Cancer Registries with Digital Pathology Specimens, Computational Imaging Tools, and Advanced Mining Capabilities

BACKGROUND

Population-based state cancer registries are an authoritative source for cancer statistics in the United States. They routinely collect a variety of data, including patient demographics, primary tumor site, stage at diagnosis, first course of treatment, and survival, on every cancer case that is reported across all U.S. states and territories. The goal of our project is to enrich NCI's Surveillance, Epidemiology, and End Results (SEER) registry data with high-quality population-based biospecimen data in the form of digital pathology, machine-learning-based classifications, and quantitative histopathology imaging feature sets (referred to here as Pathomics features).

MATERIALS AND METHODS

As part of the project, the underlying informatics infrastructure was designed, tested, and implemented through close collaboration with several participating SEER registries to ensure consistency with registry processes, computational scalability, and ability to support creation of population cohorts that span multiple sites. Utilizing computational imaging algorithms and methods to both generate indices and search for matches makes it possible to reduce inter- and intra-observer inconsistencies and to improve the objectivity with which large image repositories are interrogated.

RESULTS

Our team has created and continues to expand a well-curated repository of high-quality digitized pathology images corresponding to subjects whose data are routinely collected by the collaborating registries. Our team has systematically deployed and tested key, visual analytic methods to facilitate automated creation of population cohorts for epidemiological studies and tools to support visualization of feature clusters and evaluation of whole-slide images. As part of these efforts, we are developing and optimizing advanced search and matching algorithms to facilitate automated, content-based retrieval of digitized specimens based on their underlying image features and staining characteristics.

CONCLUSION

To meet the challenges of this project, we established the analytic pipelines, methods, and workflows to support the expansion and management of a growing repository of high-quality digitized pathology and information-rich, population cohorts containing objective imaging and clinical attributes to facilitate studies that seek to discriminate among different subtypes of disease, stratify patient populations, and perform comparisons of tumor characteristics within and across patient cohorts. We have also successfully developed a suite of tools based on a deep-learning method to perform quantitative characterizations of tumor regions, assess infiltrating lymphocyte distributions, and generate objective nuclear feature measurements. As part of these efforts, our team has implemented reliable methods that enable investigators to systematically search through large repositories to automatically retrieve digitized pathology specimens and correlated clinical data based on their computational signatures.

Chest X-ray image phase features for improved diagnosis of COVID-19 using convolutional neural network

Purpose:

Recently, the outbreak of the novel coronavirus disease 2019 (COVID-19) pandemic has seriously endangered human health and life. In fighting against COVID-19, effective diagnosis of infected patient is critical for preventing the spread of diseases. Due to limited availability of test kits, the need for auxiliary diagnostic approach has increased. Recent research has shown radiography of COVID-19 patient, such as CT and X-ray, contains salient information about the COVID-19 virus and could be used as an alternative diagnosis method. Chest X-ray (CXR) due to its faster imaging time, wide availability, low cost, and portability gains much attention and becomes very promising. In order to reduce intra- and inter-observer variability, during radiological assessment, computer-aided diagnostic tools have been used in order to supplement medical decision making and subsequent management. Computational methods with high accuracy and robustness are required for rapid triaging of patients and aiding radiologist in the interpretation of the collected data.

Method:

In this study, we design a novel multi-feature convolutional neural network (CNN) architecture for multi-class improved classification of COVID-19 from CXR images. CXR images are enhanced using a local phase-based image enhancement method. The enhanced images, together with the original CXR data, are used as an input to our proposed CNN architecture. Using ablation studies, we show the effectiveness of the enhanced images in improving the diagnostic accuracy. We provide quantitative evaluation on two datasets and qualitative results for visual inspection. Quantitative evaluation is performed on data consisting of 8851 normal (healthy), 6045 pneumonia, and 3323 COVID-19 CXR scans.

Results:

In Dataset-1, our model achieves 95.57% average accuracy for a three classes classification, 99% precision, recall, and F1-scores for COVID-19 cases. For Dataset-2, we have obtained 94.44% average accuracy, and 95% precision, recall, and F1-scores for detection of COVID-19.

Conclusions:

Our proposed multi-feature-guided CNN achieves improved results compared to single-feature CNN proving the importance of the local phase-based CXR image enhancement. Future work will involve further evaluation of the proposed method on a larger-size COVID-19 dataset as they become available.

Gene expression of adipokines and adipokine receptors in the tumor microenvironment: associations of lower expression with more aggressive breast tumor features

PURPOSE

Limited epidemiologic data are available on the expression of adipokines leptin (LEP) and adiponectin (ADIPOQ) and adipokine receptors (LEPR, ADIPOR1, ADIPOR2) in the breast tumor microenvironment (TME). The associations of gene expression of these biomarkers with tumor clinicopathology are not well understood.

METHODS

NanoString multiplexed assays were used to assess the gene expression levels of LEP, LEPR, ADIPOQ, ADIPOR1, and ADIPOR2 within tumor tissues among 162 Black and 55 White women with newly diagnosed breast cancer. Multivariate mixed effects models were used to estimate associations of gene expression with breast tumor clinicopathology (overall and separately among Blacks).

RESULTS

Black race was associated with lower gene expression of LEPR (P = 0.002) and ADIPOR1 (P = 0.01). Lower LEP, LEPR, and ADIPOQ gene expression were associated with higher tumor grade (P = 0.0007, P < 0.0001, and P < 0.0001, respectively) and larger tumor size (P < 0.0001, P = 0.0005, and P < 0.0001, respectively). Lower ADIPOQ expression was associated with ER- status (P = 0.0005), and HER2-enriched (HER2-E; P = 0.0003) and triple-negative (TN; P = 0.002) subtypes. Lower ADIPOR2 expression was associated with Ki67+ status (P = 0.0002), ER- status (P < 0.0001), PR- status (P < 0.0001), and TN subtype (P = 0.0002). Associations of lower adipokine and adipokine receptor gene expression with ER-, HER2-E, and TN subtypes were confirmed using data from The Cancer Genome Atlas (P-values < 0.005).

CONCLUSION

These findings suggest that lower expression of ADIPOQ, ADIPOR2, LEP, and LEPR in the breast TME might be indicators of more aggressive breast cancer phenotypes. Validation of these findings are warranted to elucidate the role of the adipokines and adipokine receptors in long-term breast cancer prognosis.

Integrative clinical, genomics and metabolomics data analysis for mainstream precision medicine to investigate COVID-19

Despite significant scientific and medical discoveries, the genetics of novel infectious diseases like COVID-19 remains far from understanding. SARS-CoV-2 is a single-stranded RNA respiratory virus that causes COVID-19 by binding to the ACE2 receptor in the lung and other organs. Understanding its clinical presentation and metabolomic and genetic profile will lead to the discovery of diagnostic, prognostic and predictive biomarkers, which may lead to more effective medical therapy. It is important to investigate correlations and overlap between reported diagnoses of a patient with COVID-19 in clinical data with identified germline and somatic mutations, and highly expressed genes from genomics data analysis. Timely model clinical, genomics and metabolomics data to find statistical patterns across millions of features to identify underlying biological pathways, modifiable risk factors and actionable information that supports early detection and prevention of COVID-19, and development of new therapies for better patient care. Next, ensuring security reconcile noise, need to build and train machine learning prognostic models to find actionable information that supports early detection and prevention of COVID-19. Based on the myriad data, applying appropriate machine learning algorithms to stratify patients, understand scenarios, optimise decision-making, identify high-risk rare variants (including ACE2, TMPRSS2) and making medically relevant predictions. Innovative and intelligent solutions are required to improve the traditional symptom-driven practice, and allow earlier interventions using predictive diagnostics and tailor better personalised treatments, when confronted with the challenges of pandemic situations.

Immunohistochemical analysis of adipokine and adipokine receptor expression in the breast tumor microenvironment: associations of lower leptin receptor expression with estrogen receptor-negative status and triple-negative subtype

BACKGROUND

The molecular mechanisms underlying the association between increased adiposity and aggressive breast cancer phenotypes remain unclear, but likely involve the adipokines, leptin (LEP) and adiponectin (ADIPOQ), and their receptors (LEPR, ADIPOR1, ADIPOR2).

METHODS

We used immunohistochemistry (IHC) to assess LEP, LEPR, ADIPOQ, ADIPOR1, and ADIPOR2 expression in breast tumor tissue microarrays among a sample of 720 women recently diagnosed with breast cancer (540 of whom self-identified as Black). We scored IHC expression quantitatively, using digital pathology analysis. We abstracted data on tumor grade, tumor size, tumor stage, lymph node status, Ki67, estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) from pathology records, and used ER, PR, and HER2 expression data to classify breast cancer subtype. We used multivariable mixed effects models to estimate associations of IHC expression with tumor clinicopathology, in the overall sample and separately among Blacks.

RESULTS

Larger proportions of Black than White women were overweight or obese and had more aggressive tumor features. Older age, Black race, postmenopausal status, and higher body mass index were associated with higher LEPR IHC expression. In multivariable models, lower LEPR IHC expression was associated with ER-negative status and triple-negative subtype (P < 0.0001) in the overall sample and among Black women only. LEP, ADIPOQ, ADIPOR1, and ADIPOR2 IHC expression were not significantly associated with breast tumor clinicopathology.

CONCLUSIONS

Lower LEPR IHC expression within the breast tumor microenvironment might contribute mechanistically to inter-individual variation in aggressive breast cancer clinicopathology, particularly ER-negative status and triple-negative subtype.

Statistical Analysis of Survival Models Using Feature Quantification on Prostate Cancer Histopathological Images

Background

Grading of prostatic adenocarcinoma is based on the Gleason scoring system and the more recently established prognostic grade groups. Typically, prostate cancer grading is performed by pathologists based on the morphology of the tumor on hematoxylin and eosin (H and E) slides. In this study, we investigated the histopathological image features with various survival models and attempted to study their correlations.

Methods

Three texture methods (speeded-up robust features, histogram of oriented gradient, and local binary pattern) and two convolutional neural network (CNN)-based methods were applied to quantify histopathological image features. Five survival models were assessed on those image features in the context with other prostate clinical prognostic factors, including primary and secondary Gleason patterns, prostate-specific antigen levels, age, and clinical tumor stages.

Results

Based on statistical comparisons among different image features with survival models, image features from CNN-based method with a recurrent neural network called CNN-long-short-term memory provided the highest hazard ratio of prostate cancer recurrence under Cox regression with an elastic net penalty.

Conclusions

This approach outperformed the other image quantification methods listed above. Using this approach, patient outcomes were highly correlated with the histopathological image features of the tissue samples. In future studies, we plan to investigate the potential use of this approach for predicting recurrence in a wider range of cancer types.

Two well-differentiated pancreatic neuroendocrine tumor mouse models

Multiple endocrine neoplasia type 1 (MEN1) is a genetic syndrome in which patients develop neuroendocrine tumors (NETs), including pancreatic neuroendocrine tumors (PanNETs). The prolonged latency of tumor development in MEN1 patients suggests a likelihood that other mutations cooperate with Men1 to induce PanNETs. We propose that Pten loss combined with Men1 loss accelerates tumorigenesis. To test this, we developed two genetically engineered mouse models (GEMMs)-MPR (Men1^flox/flox Pten^flox/flox RIP-Cre) and MPM (Men1^flox/flox Pten^flox/flox MIP-Cre) using the Cre-LoxP system with insulin-specific biallelic inactivation of Men1 and Pten. Cre in the MPR mouse model was driven by the transgenic rat insulin 2 promoter while in the MPM mouse model was driven by the knock-in mouse insulin 1 promoter. Both mouse models developed well-differentiated (WD) G1/G2 PanNETs at a much shorter latency than Men1 or Pten single deletion alone and exhibited histopathology of human MEN1-like tumor. The MPR model, additionally, developed pituitary neuroendocrine tumors (PitNETs) in the same mouse at a much shorter latency than Men1 or Pten single deletion alone as well. Our data also demonstrate that Pten plays a role in NE tumorigenesis in pancreas and pituitary. Treatment with the mTOR inhibitor rapamycin delayed the growth of PanNETs in both MPR and MPM mice, as well as the growth of PitNETs, resulting in prolonged survival in MPR mice. Our MPR and MPM mouse models are the first to underscore the cooperative roles of Men1 and Pten in cancer, particularly neuroendocrine cancer. The early onset of WD PanNETs mimicking the human counterpart in MPR and MPM mice at 7 weeks provides an effective platform for evaluating therapeutic opportunities for NETs through targeting the MENIN-mediated and PI3K/AKT/mTOR signaling pathways.

Unsupervised Domain Adaptation for Classification of Histopathology Whole-Slide Images

Computational image analysis is one means for evaluating digitized histopathology specimens that can increase the reproducibility and reliability with which cancer diagnoses are rendered while simultaneously providing insight as to the underlying mechanisms of disease onset and progression. A major challenge that is confronted when analyzing samples that have been prepared at disparate laboratories and institutions is that the algorithms used to assess the digitized specimens often exhibit heterogeneous staining characteristics because of slight differences in incubation times and the protocols used to prepare the samples. Unfortunately, such variations can render a prediction model learned from one batch of specimens ineffective for characterizing an ensemble originating from another site. In this work, we propose to adopt unsupervised domain adaptation to effectively transfer the discriminative knowledge obtained from any given source domain to the target domain without requiring any additional labeling or annotation of images at the target site. In this paper, our team investigates the use of two approaches for performing the adaptation: (1) color normalization and (2) adversarial training. The adversarial training strategy is implemented through the use of convolutional neural networks to find an invariant feature space and Siamese architecture within the target domain to add a regularization that is appropriate for the entire set of whole-slide images. The adversarial adaptation results in significant classification improvement compared with the baseline models under a wide range of experimental settings.

Recurrence analysis on prostate cancer patients with Gleason score 7 using integrated histopathology whole-slide images and genomic data through deep neural networks

Prostate cancer is the most common nonskin-related cancer, affecting one in seven men in the United States. Gleason score, a sum of the primary and secondary Gleason patterns, is one of the best predictors of prostate cancer outcomes. Recently, significant progress has been made in molecular subtyping prostate cancer through the use of genomic sequencing. It has been established that prostate cancer patients presented with a Gleason score 7 show heterogeneity in both disease recurrence and survival. We built a unified system using publicly available whole-slide images and genomic data of histopathology specimens through deep neural networks to identify a set of computational biomarkers. Using a survival model, the experimental results on the public prostate dataset showed that the computational biomarkers extracted by our approach had hazard ratio as 5.73 and C -index as 0.74, which were higher than standard clinical prognostic factors and other engineered image texture features. Collectively, the results of this study highlight the important role of neural network analysis of prostate cancer and the potential of such approaches in other precision medicine applications.

Computer aided analysis of prostate histopathology images Gleason grading especially for Gleason score 7

Clinically, prostate adenocarcinoma is diagnosed by recognizing certain morphology on histology. While the Gleason grading system has been shown to be the strongest prognostic factor for men with prostrate adenocarcinoma, there is a significant intra and interobserver variability between pathologists in assigning this grading system. In this study, we present a new method for prostate gland segmentation from which we then utilize to develop a computer aided Gleason grading. The novelty of our method is a region-based nuclei segmentation to get individual gland without using lumen as prior information. Because each gland region is surrounded by nuclei, individual gland can be segmented by using the structure features and Delaunay Triangulation. The precision, recal and F1 of this approach are 0.94±0.11, 0.60±0.23 and 0.70±0.19 respectively. Our method achieves a high accuracy for prostate gland segmentation with less computation time.

Differentiation among prostate cancer patients with Gleason score of 7 using histopathology whole-slide image and genomic data

Prostate cancer is the most common non-skin related cancer affecting 1 in 7 men in the United States. Treatment of patients with prostate cancer still remains a difficult decision-making process that requires physicians to balance clinical benefits, life expectancy, comorbidities, and treatment-related side effects. Gleason score (a sum of the primary and secondary Gleason patterns) solely based on morphological prostate glandular architecture has shown as one of the best predictors of prostate cancer outcome. Significant progress has been made on molecular subtyping prostate cancer delineated through the increasing use of gene sequencing. Prostate cancer patients with Gleason score of 7 show heterogeneity in recurrence and survival outcomes. Therefore, we propose to assess the correlation between histopathology images and genomic data with disease recurrence in prostate tumors with a Gleason 7 score to identify prognostic markers. In the study, we identify image biomarkers within tissue WSIs by modeling the spatial relationship from automatically created patches as a sequence within WSI by adopting a recurrence network model, namely long short-term memory (LSTM). Our preliminary results demonstrate that integrating image biomarkers from CNN with LSTM and genomic pathway scores, is more strongly correlated with patients recurrence of disease compared to standard clinical markers and engineered image texture features. The study further demonstrates that prostate cancer patients with Gleason score of 4+3 have a higher risk of disease progression and recurrence compared to prostate cancer patients with Gleason score of 3+4.

Electromechanical Coupling Factor of Breast Tissue as a Biomarker for Breast Cancer

GOAL

This research aims to validate a new biomarker of breast cancer by introducing electromechanical coupling factor of breast tissue samples as a possible additional indicator of breast cancer. Since collagen fibril exhibits a structural organization that gives rise to a piezoelectric effect, the difference in collagen density between normal and cancerous tissue can be captured by identifying the corresponding electromechanical coupling factor.

METHODS

The design of a portable diagnostic tool and a microelectromechanical systems (MEMS)-based biochip, which is integrated with a piezoresistive sensing layer for measuring the reaction force as well as a microheater for temperature control, is introduced. To verify that electromechanical coupling factor can be used as a biomarker for breast cancer, the piezoelectric model for breast tissue is described with preliminary experimental results on five sets of normal and invasive ductal carcinoma (IDC) samples in the 25-45 temperature range.

CONCLUSION

While the stiffness of breast tissues can be captured as a representative mechanical signature which allows one to discriminate among tissue types especially in the higher strain region, the electromechanical coupling factor shows more distinct differences between the normal and IDC groups over the entire strain region than the mechanical signature. From the two-sample -test, the electromechanical coupling factor under compression shows statistically significant differences ( 0.0039) between the two groups.

SIGNIFICANCE

The increase in collagen density in breast tissue is an objective and reproducible characteristic of breast cancer. Although characterization of mechanical tissue property has been shown to be useful for differentiating cancerous tissue from normal tissue, using a single parameter may not be sufficient for practical usage due to inherent variation among biological samples. The portable breast cancer diagnostic tool reported in this manuscript shows the feasibility of measuring multiple parameters of breast tissue allowing for practical application.

Roadmap to a Comprehensive Clinical Data Warehouse for Precision Medicine Applications in Oncology

Leading institutions throughout the country have established Precision Medicine programs to support personalized treatment of patients. A cornerstone for these programs is the establishment of enterprise-wide Clinical Data Warehouses. Working shoulder-to-shoulder, a team of physicians, systems biologists, engineers, and scientists at Rutgers Cancer Institute of New Jersey have designed, developed, and implemented the Warehouse with information originating from data sources, including Electronic Medical Records, Clinical Trial Management Systems, Tumor Registries, Biospecimen Repositories, Radiology and Pathology archives, and Next Generation Sequencing services. Innovative solutions were implemented to detect and extract unstructured clinical information that was embedded in paper/text documents, including synoptic pathology reports. Supporting important precision medicine use cases, the growing Warehouse enables physicians to systematically mine and review the molecular, genomic, image-based, and correlated clinical information of patient tumors individually or as part of large cohorts to identify changes and patterns that may influence treatment decisions and potential outcomes.

Computer aided analysis of prostate histopathology images to support a refined Gleason grading system

The Gleason grading system used to render prostate cancer diagnosis has recently been updated to allow more accurate grade stratification and higher prognostic discrimination when compared to the traditional grading system. In spite of progress made in trying to standardize the grading process, there still remains approximately a 30% grading discrepancy between the score rendered by general pathologists and those provided by experts while reviewing needle biopsies for Gleason pattern 3 and 4, which accounts for more than 70% of daily prostate tissue slides at most institutions. We propose a new computational imaging method for Gleason pattern 3 and 4 classification, which better matches the newly established prostate cancer grading system. The computer-aided analysis method includes two phases. First, the boundary of each glandular region is automatically segmented using a deep convolutional neural network. Second, color, shape and texture features are extracted from superpixels corresponding to the outer and inner glandular regions and are subsequently forwarded to a random forest classifier to give a gradient score between 3 and 4 for each delineated glandular region. The F ₁ score for glandular segmentation is 0.8460 and the classification accuracy is 0.83±0.03.

Toward a Portable Cancer Diagnostic Tool Using a Disposable MEMS-Based Biochip

GOAL

The objective of this study is to design and develop a portable tool consisting of a disposable biochip for measuring electrothermomechanical (ETM) properties of breast tissue.

METHODS

A biochip integrated with a microheater, force sensors, and electrical sensors is fabricated using microtechnology. The sensor covers the area of 2 mm and the biochip is 10 mm in diameter. A portable tool capable of holding tissue and biochip is fabricated using 3-D printing. Invasive ductal carcinoma and normal tissue blocks are selected from cancer tissue bank in Biospecimen Repository Service at Rutgers Cancer Institute of New Jersey. The ETM properties of the normal and cancerous breast tissues (3-mm thickness and 2-mm diameter) are measured by indenting the tissue placed on the biochip integrated inside the 3-D printed tool.

RESULTS

Integrating microengineered biochip and 3-D printing, we have developed a portable cancer diagnosis device. Using this device, we have shown a statistically significant difference between cancerous and normal breast tissues in mechanical stiffness, electrical resistivity, and thermal conductivity.

CONCLUSION

The developed cancer diagnosis device is capable of simultaneous ETM measurements of breast tissue specimens and can be a potential candidate for delineating normal and cancerous breast tissue cores.

SIGNIFICANCE

The portable cancer diagnosis tool could potentially provide a deterministic and quantitative information about the breast tissue characteristics, as well as the onset and disease progression of the tissues. The tool can be potentially used for other tissue-related cancers.

Parallel Versus Distributed Data Access for Gigapixel-Resolution Histology Images: Challenges and Opportunities

Recent advances in digital pathology technology have led to significant improvements in terms of both the quality and resolution of the resulting images, which now often exceed several gigabytes each. Today, several leading institutions across the country utilize whole-slide imaging (WSI) as part of their routine workflow. WSIs have utility in a wide range of diagnostic and investigative pathology applications. The fact that these images are both large in size (about 30 GB when uncompressed) and are generated in nonstandard proprietary formats has limited wider adoption of these technologies and makes the task of accessing, processing, and analyzing them in high-throughput fashion extremely challenging. The common approach for such data analytic applications is to preprocess the large whole-slide images into smaller size files and store them in a generic format. However, this approach limits the advantages that might be realized if different scalability levels and data unit sizes could be dynamically changed based on the specifications of the task at hand and the architectural limits of the infrastructure (e.g., node memory size). Such strategies also introduce extra processing time to the workflow. To address these challenges, we present, in this paper, novel scalable access methods for parallel file systems and distributed file/object storage systems. Experimental results gathered during the course of our studies show that these methods provide opportunities not realizable using traditional approaches. We demonstrate tangible, scalability, and high-throughput advantages using a Lustre parallel file system and AWS S3 distributed storage system.

Scalable analysis of Big pathology image data cohorts using efficient methods and high-performance computing strategies

Background

We describe a suite of tools and methods that form a core set of capabilities for researchers and clinical investigators to evaluate multiple analytical pipelines and quantify sensitivity and variability of the results while conducting large-scale studies in investigative pathology and oncology. The overarching objective of the current investigation is to address the challenges of large data sizes and high computational demands.

Results

The proposed tools and methods take advantage of state-of-the-art parallel machines and efficient content-based image searching strategies. The content based image retrieval (CBIR) algorithms can quickly detect and retrieve image patches similar to a query patch using a hierarchical analysis approach. The analysis component based on high performance computing can carry out consensus clustering on 500,000 data points using a large shared memory system.

Conclusions

Our work demonstrates efficient CBIR algorithms and high performance computing can be leveraged for efficient analysis of large microscopy images to meet the challenges of clinically salient applications in pathology. These technologies enable researchers and clinical investigators to make more effective use of the rich informational content contained within digitized microscopy specimens.

Magnetic Resonance Imaging as a Biomarker for Renal Cell Carcinoma

As the most common neoplasm arising from the kidney, renal cell carcinoma (RCC) continues to have a significant impact on global health. Conventional cross-sectional imaging has always served an important role in the staging of RCC. However, with recent advances in imaging techniques and postprocessing analysis, magnetic resonance imaging (MRI) now has the capability to function as a diagnostic, therapeutic, and prognostic biomarker for RCC. For this narrative literature review, a PubMed search was conducted to collect the most relevant and impactful studies from our perspectives as urologic oncologists, radiologists, and computational imaging specialists. We seek to cover advanced MR imaging and image analysis techniques that may improve the management of patients with small renal mass or metastatic renal cell carcinoma.

Robot-Guided Atomic Force Microscopy for Mechano-Visual Phenotyping of Cancer Specimens

Atomic force microscopy (AFM) and other forms of scanning probe microscopy have been successfully used to assess biomechanical and bioelectrical characteristics of individual cells. When extending such approaches to heterogeneous tissue, there exists the added challenge of traversing the tissue while directing the probe to the exact location of the targeted biological components under study. Such maneuvers are extremely challenging owing to the relatively small field of view, limited availability of reliable visual cues, and lack of context. In this study we designed a system that leverages the visual topology of the serial tissue sections of interest to help guide robotic control of the AFM stage to provide the requisite navigational support. The process begins by mapping the whole-slide image of a stained specimen with a well-matched, consecutive section of unstained section of tissue in a piecewise fashion. The morphological characteristics and localization of any biomarkers in the stained section can be used to position the AFM probe in the unstained tissue at regions of interest where the AFM measurements are acquired. This general approach can be utilized in various forms of microscopy for navigation assistance in tissue specimens.

Accurate characterization of benign and cancerous breast tissues: aspecific patient studies using piezoresistive microcantilevers

Breast cancer is the largest detected cancer amongst women in the US. In this work, our team reports on the development of piezoresistive microcantilevers (PMCs) to investigate their potential use in the accurate detection and characterization of benign and diseased breast tissues by performing indentations on the micro-scale tissue specimens. The PMCs used in these experiments have been fabricated using laboratory-made silicon-on-insulator (SOI) substrate, which significantly reduces the fabrication costs. The PMCs are 260 μm long, 35 μm wide and 2 μm thick with resistivity of order 1.316×10(-3) Ω cm obtained by using boron diffusion technique. For indenting the tissue, we utilized 8 μm thick cylindrical SU-8 tip. The PMC was calibrated against a known AFM probe. Breast tissue cores from seven different specimens were indented using PMC to identify benign and cancerous tissue cores. Furthermore, field emission scanning electron microscopy (FE-SEM) of benign and cancerous specimens showed marked differences in the tissue morphology, which further validates our observed experimental data with the PMCs. While these patient aspecific feasibility studies clearly demonstrate the ability to discriminate between benign and cancerous breast tissues, further investigation is necessary to perform automated mechano-phenotyping (classification) of breast cancer: from onset to disease progression.

Robust automatic breast cancer staging using a combination of functional genomics and image-omics

Breast cancer is one of the leading cancers worldwide. Precision medicine is a new trend that systematically examines molecular and functional genomic information within each patient's cancer to identify the patterns that may affect treatment decisions and potential outcomes. As a part of precision medicine, computer-aided diagnosis enables joint analysis of functional genomic information and image from pathological images. In this paper we propose an integrated framework for breast cancer staging using image-omics and functional genomic information. The entire biomedical imaging informatics framework consists of image-omics extraction, feature combination, and classification. First, a robust automatic nuclei detection and segmentation is presented to identify tumor regions, delineate nuclei boundaries and calculate a set of image-based morphological features; next, the low dimensional image-omics is obtained through principal component analysis and is concatenated with the functional genomic features identified by a linear model. A support vector machine for differentiating stage I breast cancer from other stages are learned. We experimentally demonstrate that compared with a single type of representation (image-omics), the combination of image-omics and functional genomic feature can improve the classification accuracy by 3%.

Exploring automatic prostate histopathology image Gleason grading via local structure modeling

Gleason-grading of prostate cancer pathology specimens reveal the malignancy of the cancer tissues, thus provides critical guidance for prostate cancer diagnoses and treatment. Computer-aided automatic grading methods have been providing efficient and result-consistent alternative to traditional manually slide reading approach, through statistical and structural feature analysis of the digitized pathology slides. In this paper, we propose a novel automatic Gleason grading algorithm through local structure model learning and classification. We use attributed graph to represent the tissue glandular structures in histopathology images; representative sub-graphs features were learned as bags-of-words features from labeled samples of each grades. Then structural similarity between sub-graphs in the unlabeled images and the representative sub-graphs were obtained using the learned codebook. Gleason grade was given based on an overall similarity score. We validated the proposed algorithm on 300 prostate histopathology images from the TCGA dataset, and the algorithm achieved average grading accuracy of 91.25%, 76.36% and 64.75% on images with Gleason grade 3, 4 and 5 respectively.

Simultaneous MEMS-based electro-mechanical phenotyping of breast cancer

Carcinomas are the most commonly diagnosed cancers originating in the skin, lungs, breasts, pancreas, and other organs and glands. In most of the cases, the microenvironment within the tissue changes with the progression of disease. A key challenge is to develop a device capable of providing quantitative indicators in diagnosing cancer by measuring alteration in electrical and mechanical property of the tissues from the onset of malignancy. We demonstrate micro-electro-mechanical-systems (MEMS) based flexible polymer microsensor array capable of simultaneously measuring electro-mechanical properties of the breast tissues cores (1 mm in diameter and 10 μm in thickness) from onset through progression of the cancer. The electrical and mechanical signatures obtained from the tissue cores shows the capability of the device to clearly demarcate the specific stages of cancer in epithelial and stromal regions providing quantitative indicators facilitating the diagnosis of breast cancer. The present study shows that electro-mechanical properties of the breast tissue core at the micro-level are different than those at the macro-level.

Mechanical phenotyping of breast cancer using MEMS: a method to demarcate benign and cancerous breast tissues

The mechanical properties of tissue change significantly during the progression from healthy to malignant. Quantifying the mechanical properties of breast tissue within the tumor microenvironment can help to delineate benign from cancerous stages. In this work, we study high-grade invasive ductal carcinoma in comparison with their matched tumor adjacent areas, which exhibit benign morphology. Such paired tissue cores obtained from eight patients were indented using a MEMS-based piezoresistive microcantilever, which was positioned within pre-designated epithelial and stromal areas of the specimen. Field emission scanning electron microscopy studies on breast tissue cores were performed to understand the microstructural changes from benign to malignant. The normal epithelial tissues appeared compact and organized. The appearance of cancer regions, in comparison, not only revealed increased cellularity but also showed disorganization and increased fenestration. Using this technique, reliable discrimination between epithelial and stromal regions throughout both benign and cancerous breast tissue cores was obtained. The mechanical profiling generated using this method has the potential to be an objective, reproducible, and quantitative indicator for detecting and characterizing breast cancer.

Probabilistic estimation of mechanical properties of biomaterials using atomic force microscopy

Nanoindentation using contact-mode atomic force microscopy (AFM) has emerged as a powerful tool for effective material characterization of a wide variety of biomaterials across multiple length scales. However, the interpretation of force-indentation experimental data from AFM is subject to some debate. Uncertainties in AFM data analysis stems from two primary sources: The exact point of contact between the AFM probe and the biological specimen and the variability in the spring constant of the AFM probe. While a lot of attention has been directed toward addressing the contact-point uncertainty, the effect of variability in the probe spring constant has not received sufficient attention. In this paper, we report on an error-in-variables-based Bayesian change-point approach to quantify the elastic modulus of human breast tissue samples after accounting for variability in both contact point and the probe spring constant. We also discuss the efficacy of our approach to a wide range of hyperparameter values using a sensitivity analysis.

Content-based histopathology image retrieval using CometCloud

Background

The development of digital imaging technology is creating extraordinary levels of accuracy that provide support for improved reliability in different aspects of the image analysis, such as content-based image retrieval, image segmentation, and classification. This has dramatically increased the volume and rate at which data are generated. Together these facts make querying and sharing non-trivial and render centralized solutions unfeasible. Moreover, in many cases this data is often distributed and must be shared across multiple institutions requiring decentralized solutions. In this context, a new generation of data/information driven applications must be developed to take advantage of the national advanced cyber-infrastructure (ACI) which enable investigators to seamlessly and securely interact with information/data which is distributed across geographically disparate resources. This paper presents the development and evaluation of a novel content-based image retrieval (CBIR) framework. The methods were tested extensively using both peripheral blood smears and renal glomeruli specimens. The datasets and performance were evaluated by two pathologists to determine the concordance.

Results

The CBIR algorithms that were developed can reliably retrieve the candidate image patches exhibiting intensity and morphological characteristics that are most similar to a given query image. The methods described in this paper are able to reliably discriminate among subtle staining differences and spatial pattern distributions. By integrating a newly developed dual-similarity relevance feedback module into the CBIR framework, the CBIR results were improved substantially. By aggregating the computational power of high performance computing (HPC) and cloud resources, we demonstrated that the method can be successfully executed in minutes on the Cloud compared to weeks using standard computers.

Conclusions

In this paper, we present a set of newly developed CBIR algorithms and validate them using two different pathology applications, which are regularly evaluated in the practice of pathology. Comparative experimental results demonstrate excellent performance throughout the course of a set of systematic studies. Additionally, we present and evaluate a framework to enable the execution of these algorithms across distributed resources. We show how parallel searching of content-wise similar images in the dataset significantly reduces the overall computational time to ensure the practical utility of the proposed CBIR algorithms.

Towards an Automated MEMS-based Characterization of Benign and Cancerous Breast Tissue using Bioimpedance Measurements

Micro-Electro-Mechanical-Systems (MEMS) are desirable for use within medical diagnostics because of their capacity to manipulate and analyze biological materials at the microscale. Biosensors can be incorporated into portable lab-on-a-chip devices to quickly and reliably perform diagnostics procedure on laboratory and clinical samples. In this paper, electrical impedance-based measurements were used to distinguish between benign and cancerous breast tissues using microchips in a real-time and label-free manner. Two different microchips having inter-digited electrodes (10 µm width with 10 µm spacing and 10 µm width with 30 µm spacing) were used for measuring the impedance of breast tissues. The system employs Agilent E4980A precision impedance analyzer. The impedance magnitude and phase were collected over a frequency range of 100 Hz to 2 MHz. The benign group and cancer group showed clearly distinguishable impedance properties. At 200 kHz, the difference in impedance of benign and cancerous breast tissue was significantly higher (3110 Ω) in the case of microchips having 10 µm spacing compared to microchip having 30 µm spacing (568 Ω).

High-throughput image analysis of tumor spheroids: a user-friendly software application to measure the size of spheroids automatically and accurately

The increasing number of applications of three-dimensional (3D) tumor spheroids as an in vitro model for drug discovery requires their adaptation to large-scale screening formats in every step of a drug screen, including large-scale image analysis. Currently there is no ready-to-use and free image analysis software to meet this large-scale format. Most existing methods involve manually drawing the length and width of the imaged 3D spheroids, which is a tedious and time-consuming process. This study presents a high-throughput image analysis software application - SpheroidSizer, which measures the major and minor axial length of the imaged 3D tumor spheroids automatically and accurately; calculates the volume of each individual 3D tumor spheroid; then outputs the results in two different forms in spreadsheets for easy manipulations in the subsequent data analysis. The main advantage of this software is its powerful image analysis application that is adapted for large numbers of images. It provides high-throughput computation and quality-control workflow. The estimated time to process 1,000 images is about 15 min on a minimally configured laptop, or around 1 min on a multi-core performance workstation. The graphical user interface (GUI) is also designed for easy quality control, and users can manually override the computer results. The key method used in this software is adapted from the active contour algorithm, also known as Snakes, which is especially suitable for images with uneven illumination and noisy background that often plagues automated imaging processing in high-throughput screens. The complimentary "Manual Initialize" and "Hand Draw" tools provide the flexibility to SpheroidSizer in dealing with various types of spheroids and diverse quality images. This high-throughput image analysis software remarkably reduces labor and speeds up the analysis process. Implementing this software is beneficial for 3D tumor spheroids to become a routine in vitro model for drug screens in industry and academia.

RUNX2 is overexpressed in melanoma cells and mediates their migration and invasion

In the present study, we investigated the role of the transcription factor RUNX2 in melanomagenesis. We demonstrated that the expression of transcriptionally active RUNX2 was increased in melanoma cell lines as compared with human melanocytes. Using a melanoma tissue microarray, we showed that RUNX2 levels were higher in melanoma cells as compared with nevic melanocytes. RUNX2 knockdown in melanoma cell lines significantly decreased Focal Adhesion Kinase expression, and inhibited their cell growth, migration and invasion ability. Finally, the pro-hormone cholecalciferol reduced RUNX2 transcriptional activity and decreased migration of melanoma cells, further suggesting a role of RUNX2 in melanoma cell migration.

Parallel content-based sub-image retrieval using hierarchical searching

MOTIVATION

The capacity to systematically search through large image collections and ensembles and detect regions exhibiting similar morphological characteristics is central to pathology diagnosis. Unfortunately, the primary methods used to search digitized, whole-slide histopathology specimens are slow and prone to inter- and intra-observer variability. The central objective of this research was to design, develop, and evaluate a content-based image retrieval system to assist doctors for quick and reliable content-based comparative search of similar prostate image patches.

METHOD

Given a representative image patch (sub-image), the algorithm will return a ranked ensemble of image patches throughout the entire whole-slide histology section which exhibits the most similar morphologic characteristics. This is accomplished by first performing hierarchical searching based on a newly developed hierarchical annular histogram (HAH). The set of candidates is then further refined in the second stage of processing by computing a color histogram from eight equally divided segments within each square annular bin defined in the original HAH. A demand-driven master-worker parallelization approach is employed to speed up the searching procedure. Using this strategy, the query patch is broadcasted to all worker processes. Each worker process is dynamically assigned an image by the master process to search for and return a ranked list of similar patches in the image.

RESULTS

The algorithm was tested using digitized hematoxylin and eosin (H&E) stained prostate cancer specimens. We have achieved an excellent image retrieval performance. The recall rate within the first 40 rank retrieved image patches is ∼90%.

AVAILABILITY AND IMPLEMENTATION

Both the testing data and source code can be downloaded from http://pleiad.umdnj.edu/CBII/Bioinformatics/.

A Semi-Automated Positioning System for contact-mode Atomic Force Microscopy (AFM)

Contact mode Atomic Force Microscopy (CM-AFM) is popularly used by the biophysics community to study mechanical properties of cells cultured in petri dishes, or tissue sections fixed on microscope slides. While cells are fairly easy to locate, sampling in spatially heterogeneous tissue specimens is laborious and time-consuming at higher magnifications. Furthermore, tissue registration across multiple magnifications for AFM-based experiments is a challenging problem, suggesting the need to automate the process of AFM indentation on tissue. In this work, we have developed an image-guided micropositioning system to align the AFM probe and human breast-tissue cores in an automated manner across multiple magnifications. Our setup improves efficiency of the AFM indentation experiments considerably. Note to Practitioners: Human breast tissue is by nature heterogeneous, and in the samples we studied, epithelial tissue is formed by groups of functional breast epithelial cells that are surrounded by stromal tissue in a complex intertwined way. Therefore sampling a specific cell type on an unstained specimen is very difficult. To aid us, we use digital stained images of the same tissue annotated by a certified pathologist to identify the region of interest (ROI) at a coarse magnification and an image-guided positioning system to place the unstained tissue near the AFM probe tip. Using our setup, we could considerably reduce AFM operating time and we believe that our setup is a viable supplement to commercial AFM stages with limited X-Y range.

Automated image interpretation and computer-assisted diagnostics

Much of the difficulty in reaching consistent evaluations of radiology and pathology imaging studies arises from subjective impressions of individual observers. Developing strategies that can reliably transform complex visual observations into well-defined algorithmic procedures is an active area of exploration which can advance clinical practice, investigative research and outcome studies. The literature shows that when characterizations are based upon computer-aided analysis, objectivity, reproducibility and sensitivity improve considerably. Advanced imaging and computational tools could potentially enable investigators to detect and track subtle changes in measurable parameters leading to the discovery of novel diagnostic and prognostic clues which are not apparent by human visual inspection alone. The overarching objective of this book chapter is to provide readers with a summary of the origin, evolution and future directions for the fields of automated image interpretation and computer-assisted diagnostics. The chapter begins with a high-level overview of the fields of image processing, pattern recognition, and computer vision followed by a description of how these disciplines relate to the more comprehensive fields of computer-assisted diagnostics and image guided decision support. Throughout the remainder of the chapter we have supplied multiple illustrative examples demonstrating how recent advances and innovations in each of these areas have impacted clinical and research activities throughout pathology and radiology including high-throughput tissue microarray analysis, multi-spectral imaging, and image co-registration.

Evaluation of hepatic tumor response to yttrium-90 radioembolization therapy using texture signatures generated from contrast-enhanced CT images

RATIONALE AND OBJECTIVES

The aim of this study was to explore the use of texture features generated from liver computed tomographic (CT) datasets as potential image-based indicators of patient response to radioembolization (RE) with yttrium-90 ((90)Y) resin microspheres, an emerging locoregional therapy for advanced-stage liver cancer.

MATERIALS AND METHODS

Overall posttherapy survival and percent change in serologic tumor marker at 3 months posttherapy represent the primary clinical outcomes in this study. Thirty advanced-stage liver cancer cases (primary and metastatic) treated with RE over a 3-year period were included. Texture signatures for tumor regions, which were delineated to reveal boundaries with normal regions, were computed from pretreatment contrast-enhanced liver CT studies and evaluated for their ability to classify patient serologic response and survival.

RESULTS

A series of systematic leave-one-out cross-validation studies using soft-margin support vector machine (SVM) classifiers showed hepatic tumor texton and local binary pattern (LBP) signatures both achieve high accuracy (96%) in discriminating subjects in terms of their serologic response. The image-based indicators were also accurate in classifying subjects by survival status (80% and 93% accuracy for texton and LBP signatures, respectively).

CONCLUSIONS

Hepatic texture signatures generated from tumor regions on pretreatment triphasic CT studies were highly accurate in differentiating among subjects in terms of serologic response and survival. These image-based computational markers show promise as potential predictive tools in candidate evaluation for locoregional therapy such as RE.

The analysis of image feature robustness using cometcloud

The robustness of image features is a very important consideration in quantitative image analysis. The objective of this paper is to investigate the robustness of a range of image texture features using hematoxylin stained breast tissue microarray slides which are assessed while simulating different imaging challenges including out of focus, changes in magnification and variations in illumination, noise, compression, distortion, and rotation. We employed five texture analysis methods and tested them while introducing all of the challenges listed above. The texture features that were evaluated include co-occurrence matrix, center-symmetric auto-correlation, texture feature coding method, local binary pattern, and texton. Due to the independence of each transformation and texture descriptor, a network structured combination was proposed and deployed on the Rutgers private cloud. The experiments utilized 20 randomly selected tissue microarray cores. All the combinations of the image transformations and deformations are calculated, and the whole feature extraction procedure was completed in 70 minutes using a cloud equipped with 20 nodes. Center-symmetric auto-correlation outperforms all the other four texture descriptors but also requires the longest computational time. It is roughly 10 times slower than local binary pattern and texton. From a speed perspective, both the local binary pattern and texton features provided excellent performance for classification and content-based image retrieval.

Ranked retrieval of segmented nuclei for objective assessment of cancer gene repositioning

BACKGROUND

Correct segmentation is critical to many applications within automated microscopy image analysis. Despite the availability of advanced segmentation algorithms, variations in cell morphology, sample preparation, and acquisition settings often lead to segmentation errors. This manuscript introduces a ranked-retrieval approach using logistic regression to automate selection of accurately segmented nuclei from a set of candidate segmentations. The methodology is validated on an application of spatial gene repositioning in breast cancer cell nuclei. Gene repositioning is analyzed in patient tissue sections by labeling sequences with fluorescence in situ hybridization (FISH), followed by measurement of the relative position of each gene from the nuclear center to the nuclear periphery. This technique requires hundreds of well-segmented nuclei per sample to achieve statistical significance. Although the tissue samples in this study contain a surplus of available nuclei, automatic identification of the well-segmented subset remains a challenging task.

RESULTS

Logistic regression was applied to features extracted from candidate segmented nuclei, including nuclear shape, texture, context, and gene copy number, in order to rank objects according to the likelihood of being an accurately segmented nucleus. The method was demonstrated on a tissue microarray dataset of 43 breast cancer patients, comprising approximately 40,000 imaged nuclei in which the HES5 and FRA2 genes were labeled with FISH probes. Three trained reviewers independently classified nuclei into three classes of segmentation accuracy. In man vs. machine studies, the automated method outperformed the inter-observer agreement between reviewers, as measured by area under the receiver operating characteristic (ROC) curve. Robustness of gene position measurements to boundary inaccuracies was demonstrated by comparing 1086 manually and automatically segmented nuclei. Pearson correlation coefficients between the gene position measurements were above 0.9 (p < 0.05). A preliminary experiment was conducted to validate the ranked retrieval in a test to detect cancer. Independent manual measurement of gene positions agreed with automatic results in 21 out of 26 statistical comparisons against a pooled normal (benign) gene position distribution.

CONCLUSIONS

Accurate segmentation is necessary to automate quantitative image analysis for applications such as gene repositioning. However, due to heterogeneity within images and across different applications, no segmentation algorithm provides a satisfactory solution. Automated assessment of segmentations by ranked retrieval is capable of reducing or even eliminating the need to select segmented objects by hand and represents a significant improvement over binary classification. The method can be extended to other high-throughput applications requiring accurate detection of cells or nuclei across a range of biomedical applications.

A fast, automatic segmentation algorithm for locating and delineating touching cell boundaries in imaged histopathology

BACKGROUND

Automated analysis of imaged histopathology specimens could potentially provide support for improved reliability in detection and classification in a range of investigative and clinical cancer applications. Automated segmentation of cells in the digitized tissue microarray (TMA) is often the prerequisite for quantitative analysis. However overlapping cells usually bring significant challenges for traditional segmentation algorithms.

OBJECTIVES

In this paper, we propose a novel, automatic algorithm to separate overlapping cells in stained histology specimens acquired using bright-field RGB imaging.

METHODS

It starts by systematically identifying salient regions of interest throughout the image based upon their underlying visual content. The segmentation algorithm subsequently performs a quick, voting based seed detection. Finally, the contour of each cell is obtained using a repulsive level set deformable model using the seeds generated in the previous step. We compared the experimental results with the most current literature, and the pixel wise accuracy between human experts' annotation and those generated using the automatic segmentation algorithm.

RESULTS

The method is tested with 100 image patches which contain more than 1000 overlapping cells. The overall precision and recall of the developed algorithm is 90% and 78%, respectively. We also implement the algorithm on GPU. The parallel implementation is 22 times faster than its C/C++ sequential implementation.

CONCLUSION

The proposed segmentation algorithm can accurately detect and effectively separate each of the overlapping cells. GPU is proven to be an efficient parallel platform for overlapping cell segmentation.

Pathology Imaging Informatics for Clinical Practice and Investigative and Translational Research

Pathologists routinely interpret gross and microscopic specimens to render diagnoses and to engage in a broad spectrum of investigative research. Multiple studies have demonstrated that imaging technologies have progressed to a level at which properly digitized specimens provide sufficient quality comparable to the traditional glass slides examinations. Continued advancements in this area will have a profound impact on the manner in which pathology is conducted from this point on. Several leading institutions have already undertaken ambitious projects directed toward digitally imaging, archiving, and sharing pathology specimens. As a result of these advances, the use of informatics in diagnostic and investigative pathology applications is expanding rapidly. In addition, the advent of novel technologies such as multispectral imaging makes it possible to visualize and analyze imaged specimens using multiple wavelengths simultaneously. As these powerful technologies become increasingly accepted and adopted, the opportunities for gaining new insight into the underlying mechanisms of diseases as well as the potential for discriminating among subtypes of pathologies are growing accordingly.

Robust segmentation of overlapping cells in histopathology specimens using parallel seed detection and repulsive level set

Automated image analysis of histopathology specimens could potentially provide support for early detection and improved characterization of breast cancer. Automated segmentation of the cells comprising imaged tissue microarrays (TMAs) is a prerequisite for any subsequent quantitative analysis. Unfortunately, crowding and overlapping of cells present significant challenges for most traditional segmentation algorithms. In this paper, we propose a novel algorithm that can reliably separate touching cells in hematoxylin-stained breast TMA specimens that have been acquired using a standard RGB camera. The algorithm is composed of two steps. It begins with a fast, reliable object center localization approach that utilizes single-path voting followed by mean-shift clustering. Next, the contour of each cell is obtained using a level set algorithm based on an interactive model. We compared the experimental results with those reported in the most current literature. Finally, performance was evaluated by comparing the pixel-wise accuracy provided by human experts with that produced by the new automated segmentation algorithm. The method was systematically tested on 234 image patches exhibiting dense overlap and containing more than 2200 cells. It was also tested on whole slide images including blood smears and TMAs containing thousands of cells. Since the voting step of the seed detection algorithm is well suited for parallelization, a parallel version of the algorithm was implemented using graphic processing units (GPU) that resulted in significant speedup over the C/C++ implementation.

A data model and database for high-resolution pathology analytical image informatics

Background:

The systematic analysis of imaged pathology specimens often results in a vast amount of morphological information at both the cellular and sub-cellular scales. While microscopy scanners and computerized analysis are capable of capturing and analyzing data rapidly, microscopy image data remain underutilized in research and clinical settings. One major obstacle which tends to reduce wider adoption of these new technologies throughout the clinical and scientific communities is the challenge of managing, querying, and integrating the vast amounts of data resulting from the analysis of large digital pathology datasets. This paper presents a data model, which addresses these challenges, and demonstrates its implementation in a relational database system.

Context:

This paper describes a data model, referred to as Pathology Analytic Imaging Standards (PAIS), and a database implementation, which are designed to support the data management and query requirements of detailed characterization of micro-anatomic morphology through many interrelated analysis pipelines on whole-slide images and tissue microarrays (TMAs).

Aims:

(1) Development of a data model capable of efficiently representing and storing virtual slide related image, annotation, markup, and feature information. (2) Development of a database, based on the data model, capable of supporting queries for data retrieval based on analysis and image metadata, queries for comparison of results from different analyses, and spatial queries on segmented regions, features, and classified objects.

Settings and Design:

The work described in this paper is motivated by the challenges associated with characterization of micro-scale features for comparative and correlative analyses involving whole-slides tissue images and TMAs. Technologies for digitizing tissues have advanced significantly in the past decade. Slide scanners are capable of producing high-magnification, high-resolution images from whole slides and TMAs within several minutes. Hence, it is becoming increasingly feasible for basic, clinical, and translational research studies to produce thousands of whole-slide images. Systematic analysis of these large datasets requires efficient data management support for representing and indexing results from hundreds of interrelated analyses generating very large volumes of quantifications such as shape and texture and of classifications of the quantified features.

Materials and Methods:

We have designed a data model and a database to address the data management requirements of detailed characterization of micro-anatomic morphology through many interrelated analysis pipelines. The data model represents virtual slide related image, annotation, markup and feature information. The database supports a wide range of metadata and spatial queries on images, annotations, markups, and features.

Results:

We currently have three databases running on a Dell PowerEdge T410 server with CentOS 5.5 Linux operating system. The database server is IBM DB2 Enterprise Edition 9.7.2. The set of databases consists of 1) a TMA database containing image analysis results from 4740 cases of breast cancer, with 641 MB storage size; 2) an algorithm validation database, which stores markups and annotations from two segmentation algorithms and two parameter sets on 18 selected slides, with 66 GB storage size; and 3) an in silico brain tumor study database comprising results from 307 TCGA slides, with 365 GB storage size. The latter two databases also contain human-generated annotations and markups for regions and nuclei.

Conclusions:

Modeling and managing pathology image analysis results in a database provide immediate benefits on the value and usability of data in a research study. The database provides powerful query capabilities, which are otherwise difficult or cumbersome to support by other approaches such as programming languages. Standardized, semantic annotated data representation and interfaces also make it possible to more efficiently share image data and analysis results.

ImageMiner: a software system for comparative analysis of tissue microarrays using content-based image retrieval, high-performance computing, and grid technology

OBJECTIVE AND DESIGN

The design and implementation of ImageMiner, a software platform for performing comparative analysis of expression patterns in imaged microscopy specimens such as tissue microarrays (TMAs), is described. ImageMiner is a federated system of services that provides a reliable set of analytical and data management capabilities for investigative research applications in pathology. It provides a library of image processing methods, including automated registration, segmentation, feature extraction, and classification, all of which have been tailored, in these studies, to support TMA analysis. The system is designed to leverage high-performance computing machines so that investigators can rapidly analyze large ensembles of imaged TMA specimens. To support deployment in collaborative, multi-institutional projects, ImageMiner features grid-enabled, service-based components so that multiple instances of ImageMiner can be accessed remotely and federated.

RESULTS

The experimental evaluation shows that: (1) ImageMiner is able to support reliable detection and feature extraction of tumor regions within imaged tissues; (2) images and analysis results managed in ImageMiner can be searched for and retrieved on the basis of image-based features, classification information, and any correlated clinical data, including any metadata that have been generated to describe the specified tissue and TMA; and (3) the system is able to reduce computation time of analyses by exploiting computing clusters, which facilitates analysis of larger sets of tissue samples.

Constitutive Smad linker phosphorylation in melanoma: a mechanism of resistance to transforming growth factor-β-mediated growth inhibition

Melanoma cells are resistant to transforming growth factor-β (TGFβ)-induced cell-cycle arrest. In this study, we investigated a mechanism of resistance involving a regulatory domain, called linker region, in Smad2 and Smad3, main downstream effectors of TGFβ. Melanoma cells in culture and tumor samples exhibited constitutive Smad2 and Smad3 linker phosphorylation. Treatment of melanoma cells with the MEK1/2 inhibitor, U0126, or the two pan-CDK and GSK3 inhibitors, Flavopiridol and R547, resulted in decreased linker phosphorylation of Smad2 and Smad3. Overexpression of the linker phosphorylation-resistant Smad3 EPSM mutant in melanoma cells resulted in an increase in expression of p15(INK4B) and p21(WAF1) , as compared with cells transfected with wild-type (WT) Smad3. In addition, the cell numbers of EPSM Smad3-expressing melanoma cells were significantly reduced compared with WT Smad3-expressing cells. These results suggest that the linker phosphorylation of Smad3 contributes to the resistance of melanoma cells to TGFβ-mediated growth inhibition.

METAMERISM IN MULTISPECTRAL IMAGING OF HISTOPATHOLOGY SPECIMENS

A multispectral camera is capable of imaging a histologic slide at narrow bandwidths over the range of the visible spectrum. There is currently no clear consensus over the circumstances in which this added spectral data may improve computer-aided interpretation and diagnosis of imaged pathology specimens [1, 2, 3]. Two spectra which are perceived as the same color are called metamers, and the collection of all such spectra are referred to as the metamer set. Highly metameric colors are amenable to separation through multispectral imaging (MSI).Using the transformation between the spectrum and its perceived color, our work addresses the question of when MSI reveals information not represented by a standard RGB color image. An analytical estimate on the size of the metamer set is derived for the case of independent spectral absorption. It is shown that colors which are closest to the white point on the chromaticity diagram are highly metameric. A numerical method to estimate the metamer set in a domain-specific manner is provided. The method is demonstrated on multispectral data sets of imaged peripheral blood smears and breast tissue microarrays. An a priori estimate on the degree of metamerism from a standard color image is presented.

AN ADAPTIVE TRACKING ALGORITHM OF LUNG TUMORS IN FLUOROSCOPY USING ONLINE LEARNED COLLABORATIVE TRACKERS

Accurate tracking of tumor movement in fluoroscopic video sequences is a clinically significant and challenging problem. This is due to blurred appearance, unclear deforming shape, complicate intra- and inter- fractional motion, and other facts. Current offline tracking approaches are not adequate because they lack adaptivity and often require a large amount of manual labeling. In this paper, we present a collaborative tracking algorithm using asymmetric online boosting and adaptive appearance model. The method was applied to track the motion of lung tumors in fluoroscopic sequences provided by radiation oncologists. Our experimental results demonstrate the advantages of the method.

A multicore based parallel image registration method

Image registration is a crucial step for many image-assisted clinical applications such as surgery planning and treatment evaluation. In this paper we proposed a landmark based nonlinear image registration algorithm for matching 2D image pairs. The algorithm was shown to be effective and robust under conditions of large deformations. In landmark based registration, the most important step is establishing the correspondence among the selected landmark points. This usually requires an extensive search which is often computationally expensive. We introduced a nonregular data partition algorithm using the K-means clustering algorithm to group the landmarks based on the number of available processing cores. The step optimizes the memory usage and data transfer. We have tested our method using IBM Cell Broadband Engine (Cell/B.E.) platform.

A Parallel Point Matching Algorithm for Landmark Based Image Registration Using Multicore Platform

Point matching is crucial for many computer vision applications. Establishing the correspondence between a large number of data points is a computationally intensive process. Some point matching related applications, such as medical image registration, require real time or near real time performance if applied to critical clinical applications like image assisted surgery. In this paper, we report a new multicore platform based parallel algorithm for fast point matching in the context of landmark based medical image registration. We introduced a non-regular data partition algorithm which utilizes the K-means clustering algorithm to group the landmarks based on the number of available processing cores, which optimize the memory usage and data transfer. We have tested our method using the IBM Cell Broadband Engine (Cell/B.E.) platform. The results demonstrated a significant speed up over its sequential implementation. The proposed data partition and parallelization algorithm, though tested only on one multicore platform, is generic by its design. Therefore the parallel algorithm can be extended to other computing platforms, as well as other point matching related applications.

Investigating the Use of Cloudbursts for High-Throughput Medical Image Registration

This paper investigates the use of clouds and autonomic cloudbursting to support a medical image registration. The goal is to enable a virtual computational cloud that integrates local computational environments and public cloud services on-the-fly, and support image registration requests from different distributed researcher groups with varied computational requirements and QoS constraints. The virtual cloud essentially implements shared and coordinated task-spaces, which coordinates the scheduling of jobs submitted by a dynamic set of research groups to their local job queues. A policy-driven scheduling agent uses the QoS constraints along with performance history and the state of the resources to determine the appropriate size and mix of the public and private cloud resource that should be allocated to a specific request. The virtual computational cloud and the medical image registration service have been developed using the CometCloud engine and have been deployed on a combination of private clouds at Rutgers University and the Cancer Institute of New Jersey and Amazon EC2. An experimental evaluation is presented and demonstrates the effectiveness of autonomic cloudbursts and policy-based autonomic scheduling for this application.

A caGRID-ENABLED, LEARNING BASED IMAGE SEGMENTATION METHOD FOR HISTOPATHOLOGY SPECIMENS

Accurate segmentation of tissue microarrays is a challenging topic because of some of the similarities exhibited by normal tissue and tumor regions. Processing speed is another consideration when dealing with imaged tissue microarrays as each microscopic slide may contain hundreds of digitized tissue discs. In this paper, a fast and accurate image segmentation algorithm is presented. Both a whole disc delineation algorithm and a learning based tumor region segmentation approach which utilizes multiple scale texton histograms are introduced. The algorithm is completely automatic and computationally efficient. The mean pixel-wise segmentation accuracy is about 90%. It requires about 1 second for whole disc (1024×1024 pixels) segmentation and less than 5 seconds for segmenting tumor regions. In order to enable remote access to the algorithm and collaborative studies, an analytical service is implemented using the caGrid infrastructure. This service wraps the algorithm and provides interfaces for remote clients to submit images for analysis and retrieve analysis results.

3D SEGMENTATION OF THE LIVER USING FREE-FORM DEFORMATION BASED ON BOOSTING AND DEFORMATION GRADIENTS

This paper presents a novel automatic 3D hybrid segmentation approach based on free-form deformation. The algorithms incorporate boosting and deformation gradients to achieve reliable liver segmentation of Computed Tomography (CT) scans. A free-form deformable model is deformed under the forces originating from boosting and deformation gradients. The basic idea of the scheme is to combine information from intensity and shape prior knowledge to calculate desired displacements to the liver boundary on vertices of deformable surface. Boosting classifies the 3D image into a binary mask and the edgeflow generates a force field from the mask. The deformable surface deforms iteratively according to the force field. Deformation gradients cast restriction at each deformation step. The deformation converges to a stable status to achieve the final segmentation surface.