A Novel 29-Messenger RNA Host-Response Assay From Whole Blood Accurately Identifies Bacterial and Viral Infections in Patients Presenting to the Emergency Department With Suspected Infections: A Prospective Observational Study

OBJECTIVES

The rapid diagnosis of acute infections and sepsis remains a serious challenge. As a result of limitations in current diagnostics, guidelines recommend early antimicrobials for suspected sepsis patients to improve outcomes at a cost to antimicrobial stewardship. We aimed to develop and prospectively validate a new, 29-messenger RNA blood-based host-response classifier Inflammatix Bacterial Viral Non-Infected version 2 (IMX-BVN-2) to determine the likelihood of bacterial and viral infections.

DESIGN

Prospective observational study.

SETTING

Emergency Department, Campus Benjamin Franklin, Charité-Universitätsmedizin Berlin, Germany.

PATIENTS

Three hundred twelve adult patients presenting to the emergency department with suspected acute infections or sepsis with at least one vital sign change.

INTERVENTIONS

None (observational study only).

MEASUREMENTS AND MAIN RESULTS

Gene expression levels from extracted whole blood RNA was quantified on a NanoString nCounter SPRINT (NanoString Technologies, Seattle, WA). Two predicted probability scores for the presence of bacterial and viral infection were calculated using the IMX-BVN-2 neural network classifier, which was trained on an independent development set. The IMX-BVN-2 bacterial score showed an area under the receiver operating curve for adjudicated bacterial versus ruled out bacterial infection of 0.90 (95% CI, 0.85-0.95) compared with 0.89 (95% CI, 0.84-0.94) for procalcitonin with procalcitonin being used in the adjudication. The IMX-BVN-2 viral score area under the receiver operating curve for adjudicated versus ruled out viral infection was 0.83 (95% CI, 0.77-0.89).

CONCLUSIONS

IMX-BVN-2 demonstrated accuracy for detecting both viral infections and bacterial infections. This shows the potential of host-response tests as a novel and practical approach for determining the causes of infections, which could improve patient outcomes while upholding antimicrobial stewardship.

Optimization of Genomic Classifiers for Clinical Deployment: Evaluation of Bayesian Optimization to Select Predictive Models of Acute Infection and In-Hospital Mortality

Acute infection, if not rapidly and accurately detected, can lead to sepsis, organ failure and even death. Current detection of acute infection as well as assessment of a patient's severity of illness are imperfect. Characterization of a patient's immune response by quantifying expression levels of specific genes from blood represents a potentially more timely and precise means of accomplishing both tasks. Machine learning methods provide a platform to leverage this host response for development of deployment-ready classification models. Prioritization of promising classifiers is dependent, in part, on hyperparameter optimization for which a number of approaches including grid search, random sampling and Bayesian optimization have been shown to be effective. We compare HO approaches for the development of diagnostic classifiers of acute infection and in-hospital mortality from gene expression of 29 diagnostic markers. We take a deployment-centered approach to our comprehensive analysis, accounting for heterogeneity in our multi-study patient cohort with our choices of dataset partitioning and hyperparameter optimization objective as well as assessing selected classifiers in external (as well as internal) validation. We find that classifiers selected by Bayesian optimization for in-hospital mortality can outperform those selected by grid search or random sampling. However, in contrast to previous research: 1) Bayesian optimization is not more efficient in selecting classifiers in all instances compared to grid search or random sampling-based methods and 2) we note marginal gains in classifier performance in only specific circumstances when using a common variant of Bayesian optimization (i.e. automatic relevance determination). Our analysis highlights the need for further practical, deployment-centered benchmarking of HO approaches in the healthcare context.

Cost Impact Model of a Novel Multi-mRNA Host Response Assay for Diagnosis and Risk Assessment of Acute Respiratory Tract Infections and Sepsis in the Emergency Department

BACKGROUND

Early identification of acute infections and sepsis remains an unmet medical need. While early detection and initiation of treatment reduces mortality, inappropriate treatment leads to adverse events and the development of antimicrobial resistance. Current diagnostic and prognostic solutions, including procalcitonin, lack required accuracy. A novel blood-based host response test, HostDx™ Sepsis by Inflammatix, Inc., assesses the likelihood of a bacterial infection, the likelihood of a viral infection, and the severity of the condition.

OBJECTIVES

We estimated the economic impact of adopting HostDx Sepsis testing among patients with suspected acute respiratory tract infection (ARTI) in the emergency department (ED).

METHODS

Our cost impact model estimated costs for adult ED patients with suspected ARTI under the standard of care versus with the adoption of HostDx Sepsis from the perspective of US payers. Included costs were those assumed to be associated with an episode of sepsis diagnosis, management, and treatment. Projected accuracies for test predictions, disease prevalence, and clinical parameters was derived from patient-level meta-analysis data of randomized trials, supplemented with published performance data for HostDx Sepsis. One-way sensitivity analysis was performed on key input parameters.

RESULTS

Compared to standard of care including procalcitonin, the superior test characteristics of HostDx Sepsis resulted in an average cost savings of approximately US$1974 per patient (-31.3%) exclusive of the cost of HostDx Sepsis. Reductions in hospital days (-0.80 days, -36.7%), antibiotic days (-1.49 days, -29.5%), and percent 30-day mortality (-1.67%, -13.64%) were driven by HostDx Sepsis providing fewer "noninformative" moderate risk predictions and more "certain" low- or high-risk predictions compared to standard of care, especially for patients who were not severely ill. These results were robust to changes in key parameters, including disease prevalence.

CONCLUSIONS

Our model shows substantial savings associated with introduction of HostDx Sepsis among patients with ARTIs in EDs. These results need confirmation in interventional trials.

A generalizable 29-mRNA neural-network classifier for acute bacterial and viral infections

Improved identification of bacterial and viral infections would reduce morbidity from sepsis, reduce antibiotic overuse, and lower healthcare costs. Here, we develop a generalizable host-gene-expression-based classifier for acute bacterial and viral infections. We use training data (N = 1069) from 18 retrospective transcriptomic studies. Using only 29 preselected host mRNAs, we train a neural-network classifier with a bacterial-vs-other area under the receiver-operating characteristic curve (AUROC) 0.92 (95% CI 0.90–0.93) and a viral-vs-other AUROC 0.92 (95% CI 0.90–0.93). We then apply this classifier, inflammatix-bacterial-viral-noninfected-version 1 (IMX-BVN-1), without retraining, to an independent cohort (N = 163). In this cohort, IMX-BVN-1 AUROCs are: bacterial-vs.-other 0.86 (95% CI 0.77–0.93), and viral-vs.-other 0.85 (95% CI 0.76–0.93). In patients enrolled within 36 h of hospital admission (N = 70), IMX-BVN-1 AUROCs are: bacterial-vs.-other 0.92 (95% CI 0.83–0.99), and viral-vs.-other 0.91 (95% CI 0.82–0.98). With further study, IMX-BVN-1 could provide a tool for assessing patients with suspected infection and sepsis at hospital admission.

Diagnosing acute infections based on transcriptional host response shows promise, but generalizability is wanting. Here, the authors use a co-normalization framework to train a classifier to diagnose acute infections and apply it to independent data on a targeted diagnostic platform.

Clinical validation of an immunohistochemistry-based CanAssist-Breast test for distant recurrence prediction in hormone receptor-positive breast cancer patients

CanAssist‐Breast (CAB) is an immunohistochemistry (IHC)‐based prognostic test for early‐stage Hormone Receptor (HR+)‐positive breast cancer patients. CAB uses a Support Vector Machine (SVM) trained algorithm which utilizes expression levels of five biomarkers (CD44, ABCC4, ABCC11, N‐Cadherin, and Pan‐Cadherin) and three clinical parameters such as tumor size, grade, and node status as inputs to generate a risk score and categorizes patients as low‐ or high‐risk for distant recurrence within 5 years of diagnosis. In this study, we present clinical validation of CAB. CAB was validated using a retrospective cohort of 857 patients. All patients were treated either with endocrine therapy or chemoendocrine therapy. Risk categorization by CAB was analyzed by calculating Distant Metastasis‐Free Survival (DMFS) and recurrence rates using Kaplan‐Meier survival curves. Multivariate analysis was performed to calculate Hazard ratios (HR) for CAB high‐risk vs low‐risk patients. The results showed that Distant Metastasis‐Free Survival (DMFS) was significantly different (P‐0.002) between low‐ (DMFS: 95%) and high‐risk (DMFS: 80%) categories in the endocrine therapy treated alone subgroup (n = 195) as well as in the total cohort (n = 857, low‐risk DMFS: 95%, high‐risk DMFS: 84%, P < 0.0001). In addition, the segregation of the risk categories was significant (P = 0.0005) in node‐positive patients, with a difference in DMFS of 12%. In multivariate analysis, CAB risk score was the most significant predictor of distant recurrence with hazard ratio of 3.2048 (P < 0.0001). CAB stratified patients into discrete risk categories with high statistical significance compared to Ki‐67 and IHC4 score‐based stratification. CAB stratified a higher percentage of the cohort (82%) as low‐risk than IHC4 score (41.6%) and could re‐stratify >74% of high Ki‐67 and IHC4 score intermediate‐risk zone patients into low‐risk category. Overall the data suggest that CAB can effectively predict risk of distant recurrence with clear dichotomous high‐ or low‐risk categorization.

Development of a Novel Proteomic Risk-Classifier for Prognostication of Patients With Early-Stage Hormone Receptor-Positive Breast Cancer

Use of proteomic strategies to identify a risk classifier that estimates probability of distant recurrence in early-stage hormone receptor (HR)-positive breast cancer is relevant to physiological cellular function and therefore to intrinsic tumor biology. We used a 298-sample retrospective training set to develop an immunohistochemistry-based novel risk classifier called CanAssist-Breast (CAB) which combines 5 prognostically relevant biomarkers and 3 clinico-pathological parameters to arrive at probability of distant recurrence within 5 years from diagnosis. Five selected biomarkers, namely, CD44, ABCC4, ABCC11, N-cadherin, and pan-cadherin, were chosen based on their role in tumor metastasis. The chosen biomarkers represent the hallmarks of cancer and are distinct from other proliferation and gene expression-based prognostic signatures. The 3 clinico-pathological parameters integrated into the machine learning-based CAB algorithm are tumor size, tumor grade, and node status. These features are used to calculate a "CAB risk score" that classifies patients into low- or high-risk groups and predicts probability of distant recurrence in 5 years. Independent clinical validation of CAB in a retrospective study comprising 196 patients indicated that distant metastasis-free survival (DMFS) was significantly different in the 2 risk groups. The difference in DMFS between the low- and high-risk categories was 19% in the validation cohort (P = .0002). In multivariate analysis, CAB risk score was the most significant independent predictor of distant recurrence with a hazard ratio of 4.3 (P = .0003). CanAssist-Breast is a precise and unique machine learning-based proteomic risk-classifier that can assist in risk stratification of patients with early-stage HR+ breast cancer.

No association between donor telomere length and outcomes after allogeneic unrelated hematopoietic cell transplant in patients with acute leukemia

Recent studies suggest improved survival in patients with severe aplastic anemia receiving hematopoietic cell transplant (HCT) from unrelated donors with longer telomeres. Here, we tested whether this effect is generalizable to patients with acute leukemia. From the Center for International Blood and Marrow Transplant Research (CIBMTR®) database, we identified 1097 patients who received 8/8 HLA-matched unrelated HCT for acute myeloid leukemia (AML) or acute lymphocytic leukemia (ALL) between 2004 and 2012 with myeloablative conditioning, and had pre-HCT blood sample from the donor in CIBMTR repository. The median age at HCT for recipients was 40 years (range ≤1-68), and 32 years for donors (range = 18-61). We used qPCR for relative telomere length (RTL) measurement, and Cox proportional hazard models for statistical analyses. In a discovery cohort of 300 patients, longer donor RTL (>25th percentile) was associated with reduced risks of relapse (HR = 0.62, p = 0.05) and acute graft-versus-host disease II-IV (HR = 0.68, p = 0.05), and possibly with a higher probability of neutrophil engraftment (HR = 1.3, p = 0.06). However, these results did not replicate in two validation cohorts of 297 and 488 recipients. There was one exception; a higher probability of neutrophil engraftment was observed in one validation cohort (HR = 1.24, p = 0.05). In a combined analysis of the three cohorts, no statistically significant associations (all p > 0.1) were found between donor RTL and any outcomes.

Evaluation of a Machine Learning-Based Prognostic Model for Unrelated Hematopoietic Cell Transplantation Donor Selection

The survival of patients undergoing hematopoietic cell transplantation (HCT) from unrelated donors for acute leukemia exhibits considerable variation, even after stringent genetic matching. To improve the donor selection process, we attempted to create an algorithm to quantify the likelihood of survival to 5 years after unrelated donor HCT for acute leukemia, based on the clinical characteristics of the donor selected. All standard clinical variables were included in the model, which also included average leukocyte telomere length of the donor based on its association with recipient survival in severe aplastic anemia, and links to multiple malignancies. We developed a multivariate classifier that assigned a Preferred or NotPreferred label to each prospective donor based on the survival of the recipient. In a previous analysis using a resampling method, recipients with donors labeled Preferred experienced clinically compelling better survival compared with those labeled NotPreferred by the test. However, in a pivotal validation study in an independent cohort of 522 patients, the overall survival of the Preferred and NotPreferred donor groups was not significantly different. Although machine learning approaches have successfully modeled other biological phenomena and have led to accurate predictive models, our attempt to predict HCT outcomes after unrelated donor transplantation was not successful.

High precision prediction of functional sites in protein structures

We address the problem of assigning biological function to solved protein structures. Computational tools play a critical role in identifying potential active sites and informing screening decisions for further lab analysis. A critical parameter in the practical application of computational methods is the precision, or positive predictive value. Precision measures the level of confidence the user should have in a particular computed functional assignment. Low precision annotations lead to futile laboratory investigations and waste scarce research resources. In this paper we describe an advanced version of the protein function annotation system FEATURE, which achieved 99% precision and average recall of 95% across 20 representative functional sites. The system uses a Support Vector Machine classifier operating on the microenvironment of physicochemical features around an amino acid. We also compared performance of our method with state-of-the-art sequence-level annotator Pfam in terms of precision, recall and localization. To our knowledge, no other functional site annotator has been rigorously evaluated against these key criteria. The software and predictive models are incorporated into the WebFEATURE service at http://feature.stanford.edu/wf4.0-beta.

Abstract 1724: A gene expression profile test to resolve squamous cell carcinomas of head & neck from squamous cell carcinomas of the lung

The differential diagnosis between squamous cell carcinomas of the head & neck and squamous cell carcinomas of the lung is often unresolved because the histologic appearance of these two tumor types is quite similar. We have developed and validated a gene expression profile test (Pathwork Tissue of Origin Head & Neck Test) that distinguishes head & neck squamous and lung squamous cancers in formalin-fixed, paraffin-embedded (FFPE) specimens using a 2160-gene classification model. The test was validated in a blinded study using a pre-specified algorithm and microarray data files for 76 metastatic, poorly differentiated or undifferentiated FFPE tumor specimens that had either a known head & neck squamous or lung squamous diagnosis. The study met the primary Bayesian statistical endpoint for acceptance. Measures of test performance include overall agreement with the known diagnosis of 82.9% (95% CI, 72.5% to 90.6%), an area under the ROC curve (AUC) of 0.91 and a diagnostics odds ratio (DOR) of 23.6. Head & neck squamous cancers (N=38) gave an agreement with the known diagnosis of 81.6% and lung squamous cancers (N=38) gave an agreement of 84.2%. Reproducibility in test results between three laboratories had a concordance of 91.7%. The Tissue of Origin Head & Neck Test can aid in resolving the important differential diagnostic question for head & neck and squamous cell carcinoma of the lung.

Citation Format: {Authors}. {Abstract title} [abstract]. In: Proceedings of the 103rd Annual Meeting of the American Association for Cancer Research; 2012 Mar 31-Apr 4; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2012;72(8 Suppl):Abstract nr 1724. doi:1538-7445.AM2012-1724

Validation and reproducibility of a microarray-based gene expression test for tumor identification in formalin-fixed, paraffin-embedded specimens

Tumors whose primary site is challenging to diagnose represent a considerable proportion of new cancer cases. We present validation study results for a gene expression-based diagnostic test (the Pathwork Tissue of Origin Test) that aids in determining the tissue of origin using formalin-fixed, paraffin-embedded (FFPE) specimens. Microarray data files were generated for 462 metastatic, poorly differentiated, or undifferentiated FFPE tumor specimens, all of which had a reference diagnosis. The reference diagnoses were masked, and the microarray data files were analyzed using a 2000-gene classification model. The algorithm quantifies the similarity between RNA expression patterns of the study specimens and the 15 tissues on the test panel. Among the 462 specimens, overall agreement with the reference diagnosis was 89% (95% CI, 85% to 91%). In addition to the positive test results (ie, rule-ins), an average of 12 tissues for each specimen could be ruled out with >99% probability. The large size of this study increases confidence in the test results. A multisite reproducibility study showed 89.3% concordance between laboratories. The Tissue of Origin Test makes the benefits of microarray-based gene expression tests for tumor diagnosis available for use with the most common type of histology specimen (ie, FFPE).

Validation of a microarray-based gene expression test for tumors with uncertain origins using formalin-fixed paraffin-embedded (FFPE) specimens

e22015

Background: Microarray-based gene expression has been validated as an aid in the diagnosis of tumors with uncertain origins when the specimen is frozen tissue. Microarray use has been largely limited to RNA derived from frozen specimens. This study evaluated performance of a microarray-based test in identifying the tumor type in FFPE specimens. Methods: ZFFPE human tumor specimens (n=405) representing the 15 tissue of origin sites on the Pathwork® Tissue of Origin Test panel were blinded and evenly distributed between two independent processing labs. All specimens consisted of a 10-μm-paraffin curl containing at least 60% viable tumor and were either metastatic or poorly differentiated primaries. Each specimen was processed through RNA extraction, amplification, labeling, hybridization to a Pathchip® microarray, and was scanned to generate a qualified data file. A pre-specified classification algorithm utilizing more than 1500 genes was applied to each data file to yield Similarity Scores corresponding to the 15 tissues on the test panel. Results were then unblinded and compared to the available diagnoses. Results: Of the 405 specimens, 352 yielded qualified data files (87%). Based on the top Similarity Score, the overall agreement with available diagnoses was 89% (95% CI, 85%-92%) and for each specimen an average of 12 out of 15 tissues could be ruled out with > 99% probability. Results for all tissue types were highly informative with diagnostic odds ratios ranging from 178 to 28509. Performance was similar for metastatic (n=150; 91% agreement) and poorly differentiated primary specimens (n=202; 87% agreement). Conclusions: The large size of this study allows an accurate estimate of the confidence of test predictions for both ruling in and ruling out tissues as likely sites of primary origin. The Pathwork Tissue of Origin Test makes the potential benefits of microarray-based gene expression tests for tumors with uncertain origins available for use with the most common type of histology specimen, FFPE.

[Table: see text]

Multi-clinical laboratory validation of a gene expression-based tissue of origin test applied to poorly differentiated and undifferentiated cancers

21161

Background: This multi-clinical laboratory validation assessed the accuracy, reproducibility and robustness of the Pathwork™ Tissue of Origin Test. We focused on tumor specimens that are poorly differentiated to undifferentiated (G3-G4; high grade) because this type of specimen can consume a disproportionate amount of physician time and diagnostic resources. A subset of this type of specimen ultimately will be categorized as cancer of unknown primary (CUP), which comprises 4–5% of all malignancies. The Test uses proprietary analytics and a companion genomic microarray (Pathchip™) to compare expression levels of poorly differentiated to undifferentiated specimens with 15 types of cancer. Test development involved over 2000 specimens from 14 labs, representing 15 cancers and sixty morphologies. Methods: The studies addressed operability limits, specimen stability, potential interferences and reproducibility, as well as validation in molecular diagnostics and commercial laboratories. In all, over 700 frozen tumor specimens and microarrays were analyzed. Results: Clinical validation data demonstrated that the percent agreement of the Tissue of Origin Test was 88% across all tissue types. Between-lab concordance was 95%. If the quantitative result (Similarity Score) exceeded the recommended threshold, the probability that the indicated tissue is present was 95% across all tissue types. If the Similarity Score was less than 5, the probability that the indicated tissue is absent was 98% across all tissues types. Conclusions: The Pathwork™ Tissue of Origin Test, which is under development, is accurate and highly reproducible across laboratories in this study. The test gave consistent results for different tissue handling and extraction methods and has the potential to be an effective aid in the diagnosis of cancer patients presenting with poorly differentiated and undifferentiated tumors.

No significant financial relationships to disclose.

Gene expression analytics identify tissue of origin of unspecified metastatic cancers

20082

Background: Identification of the tissue of origin for unspecified metastatic cancers is associated with improved outcomes. To aid in determining the origin of these cancers, we designed a semi-quantitative test that uses proprietary analytics and a companion genomic microarray to compare the expression signature of a biopsy specimen with those of the following types of cancers: bladder, breast, colorectal, gastroesophageal, germ line, hepatocellular, kidney, lung, lymphoma, melanoma, ovarian, pancreatic, prostate, soft tissue-sarcoma, and thyroid. More than 5,500 human specimens processed on Affymetrix HG U133A microarrays were analyzed to determine the test’s standardization algorithm. Methods: Studies across nine different laboratories were conducted to demonstrate the reproducibility required to support the potential clinical application of the test. Biopsy specimens of 604 metastatic cancers of known origin (by conventional pathological testing) were used to “train” the algorithm. An additional 636 samples then were tested. Diagnostic odds ratio (OR), sensitivity, specificity, positive likelihood ratio (LR+), and negative likelihood ratio (LR−) were determined for individual all-against-one tests. Results: Robust standardization, which involves 121 “house-keeping genes”, enabled the tissue of origin test to demonstrate reproducibility across nine different laboratories. Identification of the correct tissue of origin was associated with OR values ranging from 23 to >500, with averages for sensitivity of 83%, specificity of 99%, LR+ of 210, and LR− of 0.17. Conclusions: The PathWork Oncology Suite: Tissue of Origin (POS:TOO) test uses methods that are reproducible across laboratories. By analyzing microarray data with the POS:TOO algorithm, the test can identify the origin of metastatic tumors. Preparations for clinical validation studies are in progress and results will be submitted to FDA for clearance.

[Table: see text]