Quantifying risk stratification provided by diagnostic tests and risk predictions: Comparison to AUC and decision curve analysis

Application of a Machine Learning-Based Classification Approach for Developing Host Protein Diagnostic Models for Infectious Disease

In recent years, infectious disease diagnosis has increasingly turned to host-centered approaches as a complement to pathogen-directed ones. The former, however, typically requires the interpretation of complex multiple biomarker datasets to arrive at an informative diagnostic outcome. This report describes a machine learning (ML)-based classification workflow that is intended as a template for researchers seeking to apply ML approaches for developing host-based infectious disease biomarker classifiers. As an example, we built a classification model that could accurately distinguish between three disease etiology classes: bacterial, viral, and normal in human sera using host protein biomarkers of known diagnostic utility. After collecting protein data from known disease samples, we trained a series of increasingly complex Auto-ML models until arriving at an optimized classifier that could differentiate viral, bacterial, and non-disease samples. Even when limited to a relatively small training set size, the model had robust diagnostic characteristics and performed well when faced with a blinded sample set. We present here a flexible approach for applying an Auto-ML-based workflow for the identification of host biomarker classifiers with diagnostic utility for infectious disease, and which can readily be adapted for multiple biomarker classes and disease states.

Artificial intelligence-based image analysis in clinical testing: lessons from cervical cancer screening

Novel screening and diagnostic tests based on artificial intelligence (AI) image recognition algorithms are proliferating. Some initial reports claim outstanding accuracy followed by disappointing lack of confirmation, including our own early work on cervical screening. This is a presentation of lessons learned, organized as a conceptual step-by-step approach to bridge the gap between the creation of an AI algorithm and clinical efficacy. The first fundamental principle is specifying rigorously what the algorithm is designed to identify and what the test is intended to measure (eg, screening, diagnostic, or prognostic). Second, designing the AI algorithm to minimize the most clinically important errors. For example, many equivocal cervical images cannot yet be labeled because the borderline between cases and controls is blurred. To avoid a misclassified case-control dichotomy, we have isolated the equivocal cases and formally included an intermediate, indeterminate class (severity order of classes: case>indeterminate>control). The third principle is evaluating AI algorithms like any other test, using clinical epidemiologic criteria. Repeatability of the algorithm at the borderline, for indeterminate images, has proven extremely informative. Distinguishing between internal and external validation is also essential. Linking the AI algorithm results to clinical risk estimation is the fourth principle. Absolute risk (not relative) is the critical metric for translating a test result into clinical use. Finally, generating risk-based guidelines for clinical use that match local resources and priorities is the last principle in our approach. We are particularly interested in applications to lower-resource settings to address health disparities. We note that similar principles apply to other domains of AI-based image analysis for medical diagnostic testing.

Imaging-Based Versus Pathologic Survival Stratifications of Diffuse Glioma According to the 2021 WHO Classification System

OBJECTIVE

Imaging-based survival stratification of patients with gliomas is important for their management, and the 2021 WHO classification system must be clinically tested. The aim of this study was to compare integrative imaging- and pathology-based methods for survival stratification of patients with diffuse glioma.

MATERIALS AND METHODS

This study included diffuse glioma cases from The Cancer Genome Atlas (training set: 141 patients) and Asan Medical Center (validation set: 131 patients). Two neuroradiologists analyzed presurgical CT and MRI to assign gliomas to five imaging-based risk subgroups (1 to 5) according to well-known imaging phenotypes (e.g., T2/FLAIR mismatch) and recategorized them into three imaging-based risk groups, according to the 2021 WHO classification: group 1 (corresponding to risk subgroup 1, indicating oligodendroglioma, isocitrate dehydrogenase [IDH]-mutant, and 1p19q-co-deleted), group 2 (risk subgroups 2 and 3, indicating astrocytoma, IDH-mutant), and group 3 (risk subgroups 4 and 5, indicating glioblastoma, IDHwt). The progression-free survival (PFS) and overall survival (OS) were estimated for each imaging risk group, subgroup, and pathological diagnosis. Time-dependent area-under-the receiver operating characteristic analysis (AUC) was used to compare the performance between imaging-based and pathology-based survival model.

RESULTS

Both OS and PFS were stratified according to the five imaging-based risk subgroups (P < 0.001) and three imaging-based risk groups (P < 0.001). The three imaging-based groups showed high performance in predicting PFS at one-year (AUC, 0.787) and five-years (AUC, 0.823), which was similar to that of the pathology-based prediction of PFS (AUC of 0.785 and 0.837). Combined with clinical predictors, the performance of the imaging-based survival model for 1- and 3-year PFS (AUC 0.813 and 0.921) was similar to that of the pathology-based survival model (AUC 0.839 and 0.889).

CONCLUSION

Imaging-based survival stratification according to the 2021 WHO classification demonstrated a performance similar to that of pathology-based survival stratification, especially in predicting PFS.

Diffuse glioma, not otherwise specified: imaging-based risk stratification achieves histomolecular-level prognostication

OBJECTIVES

To determine whether imaging-based risk stratification enables prognostication in diffuse glioma, NOS (not otherwise specified).

METHODS

Data from 220 patients classified as diffuse glioma, NOS, between January 2011 and December 2020 were retrospectively included. Two neuroradiologists analyzed pre-surgical CT and MRI to assign gliomas to the three imaging-based risk types considering well-known imaging phenotypes (e.g., T2/FLAIR mismatch). According to the 2021 World Health Organization classification, the three risk types included (1) low-risk, expecting oligodendroglioma, isocitrate dehydrogenase (IDH)-mutant, and 1p/19q-codeleted; (2) intermediate-risk, expecting astrocytoma, IDH-mutant; and (3) high-risk, expecting glioblastoma, IDH-wildtype. Progression-free survival (PFS) and overall survival (OS) were estimated for each risk type. Time-dependent receiver operating characteristic analysis using 10-fold cross-validation with 100-fold bootstrapping was used to compare the performance of an imaging-based survival model with that of a historical molecular-based survival model published in 2015, created using The Cancer Genome Archive data.

RESULTS

Prognostication according to the three imaging-based risk types was achieved for both PFS and OS (log-rank test, p < 0.001). The imaging-based survival model showed high prognostic value, with areas under the curves (AUCs) of 0.772 and 0.650 for 1-year PFS and OS, respectively, similar to the historical molecular-based survival model (AUC = 0.74 for PFS and 0.87 for OS). The imaging-based survival model achieved high long-term performance in both 3-year PFS (AUC = 0.806) and 5-year OS (AUC = 0.812).

CONCLUSION

Imaging-based risk stratification achieved histomolecular-level prognostication in diffuse glioma, NOS, and could aid in guiding patient referral for insufficient or unsuccessful molecular diagnosis.

KEY POINTS

• Three imaging-based risk types enable distinct prognostication in diffuse glioma, NOS (not otherwise specified). • The imaging-based survival model achieved similar prognostic performance as a historical molecular-based survival model. • For long-term prognostication of 3 and 5 years, the imaging-based survival model showed high performance.

Association between CHADS2, CHA2DS2-VASc, ATRIA, and Essen Stroke Risk Scores and Functional Outcomes in Acute Ischemic Stroke Patients Who Received Endovascular Thrombectomy

Background: CHADS2, CHA2DS2-VASc, ATRIA, and Essen stroke risk scores are used to estimate thromboembolism risk. We aimed to investigate the association between unfavorable outcomes and stroke risk scores in patients who received endovascular thrombectomy (EVT). Methods: This study was performed using data from a nationwide, multicenter registry to explore the selection criteria for patients who would benefit from reperfusion therapies. We calculated pre-admission CHADS2, CHA2DS2-VASc, ATRIA, and Essen scores for each patient who received EVT and compared the relationship between these scores and 3-month modified Rankin Scale (mRS) records. Results: Among the 404 patients who received EVT, 213 (52.7%) patients had unfavorable outcomes (mRS 3−6). All scores were significantly higher in patients with unfavorable outcomes than in those with favorable outcomes. Multivariable logistic regression analysis indicated that CHADS2 and the ATRIA score were positively correlated with unfavorable outcomes after adjusting for body mass index and variables with p < 0.1 in the univariable analysis (CHADS2 score: odds ratio [OR], 1.484; 95% confidence interval [CI], 1.290−1.950; p = 0.005, ATRIA score, OR, 1.128; 95% CI, 1.041−1.223; p = 0.004). Conclusions: The CHADS2 and ATRIA scores were positively correlated with unfavorable outcomes and could be used to predict unfavorable outcomes in patients who receive EVT.

The Power of Public-Private Partnership in Medical Technology Innovation: Lessons From the Development of Fda-Cleared Medical Devices for Assessment of Concussion

Given the convergence of the long and challenging development path for medical devices with the need for diagnostic capabilities for mild traumatic brain injury (mTBI/concussion), the effective role of public–private partnership (PPP) can be demonstrated to yield Food and Drug Administration (FDA) clearances and innovative product introductions. An overview of the mTBI problem and landscape was performed. A detailed situation analysis of an example of a PPP yielding an innovative product was further demonstrated. The example of PPP has led to multiple FDA clearances and product introductions in the TBI diagnostic product category where there was an urgent military and public need. Important lessons included defining the primary public and military health objective for new product introduction, the importance of the government–academia–industry PPP triad with a “collaboration towards solutions” Quality-by-Design (QbD) mindset to assure clinical validity with regulatory compliance, the development of device comparators and integration of measurements into a robust, evidence-based statistical and FDA pathway, and the utility of top-down, flexible, practical action while operating within governmental guidelines and patient safety.

A simple framework to identify optimal cost-effective risk thresholds for a single screen: Comparison to Decision Curve Analysis

Decision Curve Analysis (DCA) is a popular approach for assessing biomarkers and risk models, but does not require costs and thus cannot identify optimal risk thresholds for actions. Full decision analyses can identify optimal thresholds, but typically used methods are complex and often difficult to understand. We develop a simple framework to calculate the Incremental Net Benefit for a single-time screen as a function of costs (for tests and treatments) and effectiveness (life-years gained). We provide simple expressions for the optimal cost-effective risk-threshold and, equally importantly, for the monetary value of life-years gained associated with the risk-threshold. We consider the controversy over the risk-threshold to screen women for mutations in BRCA1/2. Importantly, most, and sometimes even all, of the thresholds identified by DCA are infeasible based on their associated dollars per life-year gained. Our simple framework facilitates sensitivity analyses to cost and effectiveness parameters. The proposed approach estimates optimal risk thresholds in a simple and transparent manner, provides intuition about which quantities are critical, and may serve as a bridge between DCA and a full decision analysis.

Liverpool Lung Project lung cancer risk stratification model: calibration and prospective validation

BACKGROUND

Early detection of lung cancer saves lives, as demonstrated by the two largest published low-dose CT screening trials. Optimal implementation depends on our ability to identify those most at risk.

METHODS

Version 2 of the Liverpool Lung Project risk score (LLPv2) was developed from case-control data in Liverpool and further adapted when applied for selection of subjects for the UK Lung Screening Trial. The objective was to produce version 3 (LLPv3) of the model, by calibration to national figures for 2017. We validated both LLPv2 and LLPv3 using questionnaire data from 75 958 individuals, followed up for lung cancer over 5 years. We validated both discrimination, using receiver operating characteristic (ROC) analysis, and absolute incidence, by comparing deciles of predicted incidence with observed incidence. We calculated proportionate difference as the percentage excess or deficit of observed cancers compared with those predicted. We also carried out Hosmer-Lemeshow tests.

RESULTS

There were 599 lung cancers diagnosed over 5 years. The discrimination of both LLPv2 and LLPv3 was significant with an area under the ROC curve of 0.81 (95% CI 0.79 to 0.82). However, LLPv2 overestimated absolute risk in the population. The proportionate difference was -58.3% (95% CI -61.6% to -54.8%), that is, the actual number of cancers was only 42% of the number predicted.In LLPv3, calibrated to national 2017 figures, the proportionate difference was -22.0% (95% CI -28.1% to -15.5%).

CONCLUSIONS

While LLPv2 and LLPv3 have the same discriminatory power, LLPv3 improves the absolute lung cancer risk prediction and should be considered for use in further UK implementation studies.

Clinical and Genetic Risk Prediction of Cognitive Impairment After Blood or Marrow Transplantation for Hematologic Malignancy

PURPOSE

Using a candidate gene approach, we tested the hypothesis that individual single nucleotide polymorphisms (SNPs) and gene-level variants are associated with cognitive impairment in patients with hematologic malignancies treated with blood or marrow transplantation (BMT) and that inclusion of these SNPs improves risk prediction beyond that offered by clinical and demographic characteristics.

PATIENTS AND METHODS

In the discovery cohort, BMT recipients underwent a standardized battery of neuropsychological tests pre-BMT and at 6 months, 1 year, 2 years, and 3 years post-BMT. Associations between 68 candidate genes and cognitive impairment were assessed using generalized estimating equation models. Elastic-Net regression was used to build Base (sociodemographic), Clinical, and Combined (Base plus Clinical plus genetic) risk prediction models of post-BMT impairment. An independent nonoverlapping cohort from the BMT Survivor Study with self-report of learning/memory problems (as identified by their health care provider) was used for model replication.

RESULTS

The discovery cohort included 277 participants (58.5% males; 68.6% non-Hispanic whites; and 46.6% allogeneic BMT recipients). Adjusting for BMT type, age at BMT, sex, race/ethnicity, and cognitive reserve, SNPs in the blood-brain barrier, telomere homeostasis, and DNA repair genes were significantly associated with cognitive impairment. Compared with the Clinical Model, the Combined Model had higher predictive power in both the discovery cohort (mean area under the receiver operating characteristic curve [AUC], 0.89; 95% CI, 0.85 to 0.93 v 0.77; 95% CI, 0.71 to 0.83; P = 1.24 × 10^-9) and the replication cohort (AUC, 0.71; 95% CI, 0.66 to 0.76 v 0.63; 95% CI, 0.57 to 0.68; P = .004).

CONCLUSION

Inclusion of candidate genetic variants enhanced the prediction of risk of post-BMT cognitive impairment beyond that offered by demographic/clinical characteristics and represents a step toward a personalized approach to managing patients at high risk for cognitive impairment after BMT.