Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition

BACKGROUND AND PURPOSE

Artificial intelligence (AI) models in radiology are frequently developed and validated using datasets from a single institution and are rarely tested on independent, external datasets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multi-center AI competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency.

MATERIALS AND METHODS

In total, 1201 anonymized, full-head NCCT clinical scans from five institutions were pooled to form the dataset. The dataset encompassed normal studies as well as pathologies including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these-task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed each model's performance. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each AI model, along with the area under the ROC curve (AUROC).

RESULTS

1177 studies were processed by four teams. The median age of patients was 62, with an interquartile range of 33. 19 teams from various academic institutions registered for the competition. Of these, four teams submitted their final results. No commercial entities participated in the competition. For task 1, AUROCs ranged from 0.49 to 0.59. For task 2, two teams completed the task with AUROC values of 0.57 and 0.52. For task 3, teams had little to no agreement with the ground truth.

CONCLUSIONS

To assess the performance of AI models in real-world clinical scenarios, we analyzed their performance in the ASFNR AI Competition. The first ASFNR Competition underscored the gap between expectation and reality; the models largely fell short in their assessments. As the integration of AI tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.ABBREVIATIONS: AI = artificial intelligence; ASFNR = American Society of Functional Neuroradiology; AUROC = area under the receiver operating characteristic curve; DICOM = Digital Imaging and Communications in Medicine; GEE = generalized estimation equation; IQR = interquartile range; NPV = negative predictive value; PPV = positive predictive value; ROC = receiver operating characteristic; TBI = traumatic brain injury.

Disparities in Hemoglobin A1c Levels in the First Year After Diagnosis Among Youths With Type 1 Diabetes Offered Continuous Glucose Monitoring

Importance

Continuous glucose monitoring (CGM) is associated with improvements in hemoglobin A1c (HbA1c) in youths with type 1 diabetes (T1D); however, youths from minoritized racial and ethnic groups and those with public insurance face greater barriers to CGM access. Early initiation of and access to CGM may reduce disparities in CGM uptake and improve diabetes outcomes.

Objective

To determine whether HbA1c decreases differed by ethnicity and insurance status among a cohort of youths newly diagnosed with T1D and provided CGM.

Design, Setting, and Participants

This cohort study used data from the Teamwork, Targets, Technology, and Tight Control (4T) study, a clinical research program that aims to initiate CGM within 1 month of T1D diagnosis. All youths with new-onset T1D diagnosed between July 25, 2018, and June 15, 2020, at Stanford Children's Hospital, a single-site, freestanding children's hospital in California, were approached to enroll in the Pilot-4T study and were followed for 12 months. Data analysis was performed and completed on June 3, 2022.

Exposures

All eligible participants were offered CGM within 1 month of diabetes diagnosis.

Main Outcomes and Measures

To assess HbA1c change over the study period, analyses were stratified by ethnicity (Hispanic vs non-Hispanic) or insurance status (public vs private) to compare the Pilot-4T cohort with a historical cohort of 272 youths diagnosed with T1D between June 1, 2014, and December 28, 2016.

Results

The Pilot-4T cohort comprised 135 youths, with a median age of 9.7 years (IQR, 6.8-12.7 years) at diagnosis. There were 71 boys (52.6%) and 64 girls (47.4%). Based on self-report, participants' race was categorized as Asian or Pacific Islander (19 [14.1%]), White (62 [45.9%]), or other race (39 [28.9%]); race was missing or not reported for 15 participants (11.1%). Participants also self-reported their ethnicity as Hispanic (29 [21.5%]) or non-Hispanic (92 [68.1%]). A total of 104 participants (77.0%) had private insurance and 31 (23.0%) had public insurance. Compared with the historical cohort, similar reductions in HbA1c at 6, 9, and 12 months postdiagnosis were observed for Hispanic individuals (estimated difference, -0.26% [95% CI, -1.05% to 0.43%], -0.60% [-1.46% to 0.21%], and -0.15% [-1.48% to 0.80%]) and non-Hispanic individuals (estimated difference, -0.27% [95% CI, -0.62% to 0.10%], -0.50% [-0.81% to -0.11%], and -0.47% [-0.91% to 0.06%]) in the Pilot-4T cohort. Similar reductions in HbA1c at 6, 9, and 12 months postdiagnosis were also observed for publicly insured individuals (estimated difference, -0.52% [95% CI, -1.22% to 0.15%], -0.38% [-1.26% to 0.33%], and -0.57% [-2.08% to 0.74%]) and privately insured individuals (estimated difference, -0.34% [95% CI, -0.67% to 0.03%], -0.57% [-0.85% to -0.26%], and -0.43% [-0.85% to 0.01%]) in the Pilot-4T cohort. Hispanic youths in the Pilot-4T cohort had higher HbA1c at 6, 9, and 12 months postdiagnosis than non-Hispanic youths (estimated difference, 0.28% [95% CI, -0.46% to 0.86%], 0.63% [0.02% to 1.20%], and 1.39% [0.37% to 1.96%]), as did publicly insured youths compared with privately insured youths (estimated difference, 0.39% [95% CI, -0.23% to 0.99%], 0.95% [0.28% to 1.45%], and 1.16% [-0.09% to 2.13%]).

Conclusions and Relevance

The findings of this cohort study suggest that CGM initiation soon after diagnosis is associated with similar improvements in HbA1c for Hispanic and non-Hispanic youths as well as for publicly and privately insured youths. These results further suggest that equitable access to CGM soon after T1D diagnosis may be a first step to improve HbA1c for all youths but is unlikely to eliminate disparities entirely.

Trial Registration

ClinicalTrials.gov Identifier: NCT04336969.

Accuracy of head computed tomography scoring systems in predicting outcomes for patients with moderate to severe traumatic brain injury: A ProTECT III ancillary study

BACKGROUND

Several types of head CT classification systems have been developed to prognosticate and stratify TBI patients.

OBJECTIVE

The purpose of our study was to compare the predictive value and accuracy of the different CT scoring systems, including the Marshall, Rotterdam, Stockholm, Helsinki, and NIRIS systems, to inform specific patient management actions, using the ProTECT III population of patients with moderate to severe acute traumatic brain injury (TBI).

METHODS

We used the data collected in the patients with moderate to severe (GCS score of 4-12) TBI enrolled in the ProTECT III clinical trial. ProTECT III was a NIH-funded, prospective, multicenter, randomized, double-blind, placebo-controlled clinical trial designed to determine the efficacy of early administration of IV progesterone. The CT scoring systems listed above were applied to the baseline CT scans obtained in the trial. We assessed the predictive accuracy of these scoring systems with respect to Glasgow Outcome Scale-Extended at 6 months, disability rating scale score, and mortality.

RESULTS

A total of 882 subjects were enrolled in ProTECT III. Worse scores for each head CT scoring systems were highly correlated with unfavorable outcome, disability outcome, and mortality. The NIRIS classification was more strongly correlated than the Stockholm and Rotterdam CT scores, followed by the Helsinki and Marshall CT classification. The highest correlation was observed between NIRIS and mortality (estimated odds ratios of 4.83).

CONCLUSION

All scores were highly associated with 6-month unfavorable, disability and mortality outcomes. NIRIS was also accurate in predicting TBI patients' management and disposition.

Predictive Model to Guide Brain Magnetic Resonance Imaging Surveillance in Patients With Metastatic Lung Cancer: Impact on Real-World Outcomes

PURPOSE

Brain metastasis is common in lung cancer, and treatment of brain metastasis can lead to significant morbidity. Although early detection of brain metastasis may improve outcomes, there are no prediction models to identify high-risk patients for brain magnetic resonance imaging (MRI) surveillance. Our goal is to develop a machine learning-based clinicogenomic prediction model to estimate patient-level brain metastasis risk.

METHODS

A penalized regression competing risk model was developed using 330 patients diagnosed with lung cancer between January 2014 and June 2019 and followed through June 2021 at Stanford HealthCare. The main outcome was time from the diagnosis of distant metastatic disease to the development of brain metastasis, death, or censoring.

RESULTS

Among the 330 patients, 84 (25%) developed brain metastasis over 627 person-years, with a 1-year cumulative brain metastasis incidence of 10.2% (95% CI, 6.8 to 13.6). Features selected for model inclusion were histology, cancer stage, age at diagnosis, primary site, and RB1 and ALK alterations. The prediction model yielded high discrimination (area under the curve 0.75). When the cohort was stratified by risk using a 1-year risk threshold of > 14.2% (85th percentile), the high-risk group had increased 1-year cumulative incidence of brain metastasis versus the low-risk group (30.8% v 6.1%, P < .01). Of 48 high-risk patients, 24 developed brain metastasis, and of these, 12 patients had brain metastasis detected more than 7 months after last brain MRI. Patients who missed this 7-month window had larger brain metastases (58% v 33% largest diameter > 10 mm; odds ratio, 2.80, CI, 0.51 to 13) versus those who had MRIs more frequently.

CONCLUSION

The proposed model can identify high-risk patients, who may benefit from more intensive brain MRI surveillance to reduce morbidity of subsequent treatment through early detection.

Association between Blood and CT Imaging Biomarkers in a Cohort of Mild Traumatic Brain Injury Patients

OBJECTIVE

To analyze the relationships between traumatic brain injury (TBI) on CT imaging and blood concentration of glial fibrillary acidic protein (GFAP), ubiquitin C-terminal hydrolase-L1 (UCH-L1), and S100B.

METHODS

This prospective cohort study involved 644 TBI patients referred to Stanford Hospital's Emergency Department between November 2015 and April 2017. Plasma and serum samples of 462 patients were analyzed for levels of GFAP, UCH-L1 and S100B. Glial neuronal ratio (GNR) was calculated as the ratio between GFAP and UCH-L1 concentrations. Admission head CT scans were reviewed for TBI imaging common data elements, and performance of biomarkers for identifying TBI was assessed via area under the receiver operating characteristic curve (ROC). We also dichotomized biomarkers at established thresholds and estimated standard measures of classification accuracy. We assessed the ability of GFAP, UCH-L1 and GNR to discriminate small and large/diffuse lesions based on CT imaging using an ROC analysis.

RESULTS

In our cohort of mostly mild TBI patients, GFAP was significantly more accurate in detecting all types of acute brain injuries than UCH-L1 in terms of area under the curves (AUC) values (P<0.001), and also compared to S100B (P<0.001). UCH-L1 and S100B had similar performance (comparable AUC values, P=0.342). Sensitivity exceeded 0.8 for each biomarker across all different types of TBI injuries, and no significant differences were observed by type of injury. Significant differences of GFAP and GNR distinguished between small lesions and large/diffuse lesions in all injuries (P=0.004, P=0.007).

CONCLUSIONS

GFAP, UCH-L1, and S100B show high sensitivity and negative predictive values for all types of TBI lesions on head CT. A combination of negative blood biomarkers (GFAP and UCH-L1) in a patient suspected of TBI may be used to safely obviate the need for a head CT scan. GFAP is a promising indicator to discriminate between small and large/diffuse TBI lesions.

Efficacy of a centralized, blended electronic, and human intervention to improve direct oral anticoagulant adherence: Smartphones to improve rivaroxaban ADHEREnce in atrial fibrillation (SmartADHERE) a randomized clinical trial

BACKGROUND

Improving adherence to direct oral anticoagulants (DOAC) is challenging, and simple text messaging reminders have not been effective.

METHODS

SmartADHERE was a randomized trial that tested a personalized digital and human direct oral anticoagulant adherence intervention compared to usual care. Eligibility required age ≥ 18, newly-prescribed (≤90 days) rivaroxaban for atrial fibrillation (AF), 1 of 4 at-risk criteria for nonadherence, and a smartphone. The intervention consisted of combination of a medication management smartphone app, daily app-based reminders, adaptive text messaging, and phone-based counseling for severe nonadherence. The primary outcome was the proportion of days covered by rivaroxaban (PDC) at 6 months. There were 25 U.S. sites, all cardiology and electrophysiology outpatient practices, activated for a target sample size of 378, but the study was terminated by the sponsor prior to reaching target enrollment.

RESULTS

There were 139 participants (age 65±9.6 years, 30% female, median CHA₂DS₂-VASc score 3 with IQR 2 to 4, mean total medication burden 7.7±4.4). DOAC adherence was high in both arms with no difference in the primary outcome (PDC 0.86±0.25 intervention vs 0.88±0.25 control, p=0.62) or in secondary outcomes including PDC ≥ 0.80 and medication persistence. Per protocol analyses had similar results. Because of the high overall PDC, the likelihood to answer the primary hypothesis was only 51% even if target enrollment were achieved. There were no study-related adverse events.

CONCLUSIONS

The use of a centralized digital and human adherence intervention was feasible across multiple sites. Overall adherence was much higher than expected despite prescreening for at-risk individuals. SmartADHERE illustrates the challenges of trials of behavioral and technology interventions, where enrollment itself may lead to selection bias or treatment effects. Pragmatic study designs, such as cluster randomization or stepped-wedge implementation, should be considered to improve enrollment and generalizability.

Prediction of Clinical Outcome in Patients with Large-Vessel Acute Ischemic Stroke: Performance of Machine Learning versus SPAN-100

BACKGROUND AND PURPOSE

Traditional statistical models and pretreatment scoring systems have been used to predict the outcome for acute ischemic stroke patients (AIS). Our aim was to select the most relevant features in terms of outcome prediction on the basis of machine learning algorithms for patients with acute ischemic stroke and to compare the performance between multiple models and the Stroke Prognostication Using Age and National Institutes of Health Stroke Scale (SPAN-100) index model.

MATERIALS AND METHODS

A retrospective multicenter cohort of 1431 patients with acute ischemic stroke was subdivided into recanalized and nonrecanalized patients. Extreme Gradient Boosting machine learning models were built to predict the mRS score at 90 days using clinical, imaging, combined, and best-performing features. Feature selection was performed using the relative weight and frequency of occurrence in the models. The model with the best performance was compared with the SPAN-100 index model using area under the receiver operating curve analysis.

RESULTS

In 3 groups of patients, the baseline NIHSS was the most significant predictor of outcome among all the parameters, with relative weights of 0.36∼0.69; ischemic core volume on CTP ranked as the most important imaging biomarker with relative weights of 0.29∼0.47. The model with the best-performing features had a better performance than the other machine learning models. The area under the curve of the model with the best-performing features was higher than SPAN-100 model and reached statistical significance for the total (P < .05) and the nonrecanalized patients (P < .001).

CONCLUSIONS

Machine learning-based feature selection can identify parameters with higher performance in outcome prediction. Machine learning models with the best-performing features, especially advanced CTP data, had superior performance of the recovery outcome prediction for patients with stroke at admission in comparison with SPAN-100.

The use of prothrombin complex concentrate as a warfarin reversal agent in pediatric patients undergoing orthotopic heart transplantation

BACKGROUND

Patients supported with a ventricular assist device are predisposed to severe bleeding at the time of orthotopic heart transplant due to several risk factors including anticoagulation with vitamin K antagonists. Kcentra, a four-factor prothrombin complex concentrate, has been approved by the FDA for warfarin reversal in adults prior to urgent surgery. There is a lack of published data on the preoperative use of four-factor prothrombin complex concentrates in pediatric patients undergoing cardiacsurgery.

METHODS

This is a single-center retrospective analysis of pediatric patients with a continuous-flow ventricular assist device who underwent heart transplant, comparing patients who received Kcentra for anticoagulation reversal with a historical patient cohort who did not. Consecutive patients from January 2013 to December 2017 were analyzed. The primary outcome was volume of blood product transfusion prior to cardiopulmonary bypass initiation. Secondary outcomes include blood product transfusion after cardiopulmonary bypass intraoperatively and up to 24 hours postoperatively, chest tube output within 24 hours of surgery, time to extubation, incidence of thromboembolism, and post-transplant length ofstay.

RESULTS

From 2013 to 2017, 31 patients with continuous-flow ventricular assist devices underwent heart transplant, with 27 patients included in the analysis. Fifteen patients received Kcentra compared with 12 patients who received fresh-frozen plasma for anticoagulation reversal. Compared with the control group, patients who received Kcentra had less packed red blood cells, fresh-frozen plasma, and platelets transfused prior to cardiopulmonary bypass initiation. The Kcentra group also received less packed red blood cells on bypass and less packed red blood cells after cardiopulmonary bypass termination. There were no differences in chest tube output, time to extubation, intensive care unit length of stay, or overall hospital length of stay. Neither group had thromboembolic complications detected during the first seven postoperative days.

CONCLUSION

This small retrospective study indicates that preoperative warfarin reversal with Kcentra reduces blood product exposure in pediatric patients with ventricular assist devices undergoing heart transplant.

Factors influencing infarct growth including collateral status assessed using computed tomography in acute stroke patients with large artery occlusion

In major ischemic stroke caused by a large artery occlusion, neuronal loss varies considerably across individuals without revascularization. This study aims to identify which patient characteristics are most highly associated with this variability. Demographic and clinical information were retrospectively collected on a registry of 878 patients. Imaging biomarkers including Alberta Stroke Program Early CT score, noncontrast head computed tomography infarct volume, perfusion computed tomography infarct core and penumbra, occlusion site, collateral score, and recanalization status were evaluated on the baseline and early follow-up computed tomography images. Infarct growth rates were calculated by dividing infarct volumes by the time elapsed between the computed tomography scan and the symptom onset. Collateral score was graded into four levels (0, 1, 2, and 3) in comparison with the normal side. Correlation of perfusion computed tomography and noncontrast head computed tomography infarct volumes and infarct growth rates were estimated with the nonparametric Spearman's rank correlation. Conditional inference trees were used to identify the clinical and imaging biomarkers that were most highly associated with the infarct growth rate and modified Rankin Scale at 90 days. Two hundred and thirty-two patients met the inclusion criteria for this study. The median infarct growth rates for perfusion computed tomography and noncontrast head computed tomography were 11.2 and 6.2 ml/log(min) in logarithmic model, and 18.9 and 10.4 ml/h in linear model, respectively. Noncontrast head computed tomography and perfusion computed tomography infarct volumes and infarct growth rates were significantly correlated (rho=0.53; P < 0.001). Collateral status was the strongest predictor for infarct growth rates. For collateral=0, the perfusion computed tomography and noncontrast head computed tomography infarct growth rate were 31.56 and 16.86 ml/log(min), respectively. Patients who had collateral >0 and penumbra volumes>92 ml had the lowest predicted perfusion computed tomography infarct growth rates (6.61 ml/log(min)). Collateral status was closely related to the diversity of infarct growth rates, poor collaterals were associated with a faster infarct growth rates and vice versa.

Assessing the Relationship Between American Heart Association Atherosclerotic Cardiovascular Disease Risk Score and Coronary Artery Imaging Findings

OBJECTIVE

The aim of this study was to characterize the relationship between computed tomography angiography imaging characteristics of coronary artery and atherosclerotic cardiovascular disease (ASCVD) score.

METHODS

We retrospectively identified all patients who underwent a coronary computed tomography angiography at our institution from December 2013 to July 2016, then we calculated the 10-year ASCVD score. We characterized the relationship between coronary artery imaging findings and ASCVD risk score.

RESULTS

One hundred fifty-one patients met our inclusion criteria. Patients with a 10-year ASCVD score of 7.5% or greater had significantly more arterial segments showing stenosis (46.4%, P = 0.008) and significantly higher maximal plaque thickness (1.25 vs 0.53, P = 0.001). However, among 56 patients with a 10-year ASCVD score of 7.5% or greater, 30 (53.6%) had no arterial stenosis. Furthermore, among the patients with a 10-year ASCVD score of less than 7.5%, 24 (25.3%) had some arterial stenosis.

CONCLUSIONS

There is some concordance but not a perfect overlap between 10-year ASCVD risk scores and coronary artery imaging findings.

Assessing the Relationship between Atherosclerotic Cardiovascular Disease Risk Score and Carotid Artery Imaging Findings

BACKGROUND AND PURPOSE

To characterize the relationship between computed tomography angiography (CTA) imaging characteristics of carotid artery and the 10-year risk of atherosclerotic cardiovascular disease (ASCVD) score.

METHODS

We retrospectively identified all patients who underwent a cervical CTA at our institution from January 2013 to July 2016, extracted clinical information, and calculated the 10-year ASCVD score using the Pooled Cohort Equations from the 2013 ACC/AHA guidelines. We compared the imaging features of artery atherosclerosis derived from the CTAs between low and high risk.

RESULTS

One hundred forty-six patients met our inclusion criteria. Patients with an ASCVD score ≥7.5% (64.4%) had significantly more arterial stenosis than patients with an ASCVD score <7.5% (35.6%, P < .001). Maximal plaque thickness was significantly higher (mean 2.33 vs. .42 mm, P < .001) and soft plaques (55.3% vs. 13.5%, P < .001) were significantly more frequent in patients with an ASCVD score ≥7.5%. However, among patients with a 10-year ASCVD score ≥7.5%, 33 (35.1%) had no arterial stenosis, 35 (37.2%) had a maximal plaque thickness less than. 9 mm, and 42 (44.7%) had no soft plaque. Furthermore, among the patients with a 10-year ASCVD score <7.5%, 8 (15.4%) had some arterial stenosis, 8 (15.4%) had a maximal plaque thickness more than. 9 mm, and 7 (13.5%) had soft plaque.

CONCLUSION

There is some concordance but not a perfect overlap between the 10-year ASCVD risk scores calculated from clinical and blood assessment and carotid artery imaging findings.

Impact of CyberKnife Radiosurgery on Median Overall Survival of Various Parameters in Patients with 1-12 Brain Metastases

Introduction

This study’s objective is to assess various patient, tumor and imaging characteristics and to compare median overall survival (OS) of 150 patients with 1-12 brain metastases post-CyberKnife radiosurgery (CKRS) (Accuray, Sunnyvale, California) alone.

Methods

Charts of 150 patients, from 2009-2014, treated with only CKRS for brain metastases were reviewed retrospectively for patient, tumor, and imaging characteristics. Parameters included demographics, Eastern Cooperative Oncology Group (ECOG) performance scores, number and control of extracranial disease (ECD) sites, cause of death (COD), histology, tumor volume (TV), and post-CKRS whole brain radiotherapy (WBRT). The imaging characteristics assessed were time of complete response (CR), partial response (PR), stable imaging or local failure (LF), and distal brain failure (DBF). The primary tumor Ki-67s of the breast carcinoma brain metastasis patients, who had the longest median OS of any group, were recorded when available.

Results

The predominant age group for the 150-patient cohort was the younger 17-65 years of age category, which was represented by 94 (62.7%). The 150-patient group had slightly more males, 79 (52.7%). The majority of 111 (74%) patients had an ECOG score of 1, 39 (26%) had 1 ECD site and uncontrolled ECD occurred in 112 (74.7%). The main COD was ECD in 106 (70.7%). The prevalent tumor histology was non-small cell lung carcinoma (88 of 150, 58.7%). The most common TV was 0-0.5 ccs (48 of 150, 32%). The majority of 125 (83.3%) patients did not undergo post-CKRS WBRT. Imaging outcomes were local control (LC) (CR, PR, or stable imaging) in 119 (79.3%), of whom 38 (25.3%) had CR, 56 (37.3%) PR and 25 (16.7%) stable imaging; LF was the outcome in 31 (20.7%) and DBF occured in 83 (55.3%). The median OS was 13 months. Patients 17-65 years of age had a median OS of 13 months, while those 66-88 years, had 12 months. Females versus males had median OS of 15 versus 12 months. The most prolonged median OS of 21.5 months occurred in those with an ECOG score of 0. Patients with two ECD sites had a median OS of 14.5 months, while those with controlled ECD, 20.5 months. Patients with breast cancer brain metastases had the longest median OS of 23 months. The median OS for each of three (0-0.5 ccs, 0.6-1.5 ccs, 1.6-4.0 ccs) of four CKRS TV quartiles was 13 months and for those with 4.1-28.5 ccs, 10 months. Median OS for patients with versus without post-CKRS WBRT was 23 versus 12 months. The longest median OS of 18.5 months for post-CKRS imaging outcomes was in patients with CR; those with LF had a median OS of 11.5 months. Of nine patients with breast carcinoma brain metastases with available Ki-67s from primary tumor resections, the Ki-67 values were ≥ 34% for four patients with CR, PR and stable imaging outcomes, and < 34% for five patients with LF.

Conclusions

An ECOG score of 0, ECD control, breast carcinoma brain metastasis histology. undergoing WBRT post-CKRS and CR imaging outcomes, each resulted in a longer median OS. The Ki-67 proliferation indices from primary breast carcinoma resection correlated well with the brain imaging outcomes in a small preliminary study in the present study's breast carcinoma patients with brain metastases.

Impact of CyberKnife Radiosurgery on Overall Survival and Various Parameters of Patients with 1-3 versus ≥ 4 Brain Metastases

Introduction

This study’s objective is to compare the overall survivals (OSs) and various parameters of patients with 1-3 versus ≥ 4 brain metastases post-CyberKnife radiosurgery (CKRS) (Accuray, Sunnyvale, California) alone.

Methods

Charts of 150 patients, from 2009-2014, treated with only CKRS for brain metastases were reviewed retrospectively for overall survival (OS) and patient, tumor, and imaging characteristics. Parameters included demographics, Eastern Cooperative Oncology Group (ECOG) performance scores, number and control of extracranial disease (ECD) sites, cause of death (COD), histology, tumor volume (TV), and post-CKRS whole brain radiotherapy (WBRT). The imaging characteristics assessed were time of complete response (CR), partial response (PR), stable imaging or local failure (LF), and distal brain failure (DBF). Patients and their data were divided into those with 1-3 (group 1) versus ≥ 4 brain metastases (group 2). For each CR and LF patient, absolute neutrophil count (ANC), absolute lymphocyte count (ALC)), and ANC/ALC ratio (NLR) were obtained, when available, at the time of CKRS.

Results

Both group 1 and group 2 had a median OS of 13 months. The patient median age for the 115 group 1 patients versus the 35 group 2 patients was 62 versus 56 years. Group 1 had slightly more males and group 2, females. The predominant ECOG score for each group was 1 and the number of ECD sites was one and two, respectively. Uncontrolled ECD occurred in the majority of both group 1 and group 2 patients. The main COD was ECD in both groups. The prevalent tumor histology for groups 1 and 2 was non-small cell lung carcinoma. Median TVs were 1.08 cc versus 1.42 cc for groups 1 and 2, respectively. The majority of patients in both groups did not undergo post-CKRS WBRT. Imaging outcomes were LC (CR, PR, or stable imaging) in 93 (80.9%) and 26 (74.3%) group 1 and 2 patients, of whom 32 (27.8%) and six (17.1%) had CR; 38 (33.0%) and 18 (51.4%), PR and 23 (20.0%) and two (5.7%), stable imaging; LF was the outcome in 22 (19.1%) and nine (25.7%) patients, and DBF occurred in 62 (53.9%) and 21 (60.0%), respectively. Uni- and multivariable analyses showed the independent parameters of a lower ECOG score, a greater number of ECD sites and uncontrolled ECD were significantly associated with greater mortality risk with and without accounting for other covariates. At CKRS, 19 group 1 and 2 CR patients had a mean ANC of 5.88 K/µL and a mean ALC of 1.31 K/µL and 13 (68%) of 19 had NLRs ≤ five, while 11 with LFs had a mean ANC of 5.22 K/µL and a mean ALC of 0.93 K/µL and seven (64%) had NLRs > five. An NLR ≤ five and high ALC was associated with a CR and an NLR > five and a low ALC with an LF.

Conclusions

Median OS post-CKRS was 13 months for both patients with 1-3 brain metastases and with ≥ 4. This is the only study in the literature to evaluate OS in patients with 1-3 and ≥ 4 brain metastases who were treated with CKRS alone. For groups 1 and 2 patients combined, 119 (79.3%) had LC and 38 (25.3%) had CR. The ANC, ALC, and NLR values are likely predictive of CR and LF outcomes

Abstract TP211: Plasma Cytokine Levels Differ Between Primary and Syndromic Moyamoya and Pediatric and Adult Moyamoya Disease

Background: Moyamoya disease (MMD) is a progressive intracranial arteriopathy of unknown etiology affecting adults and children. We aimed to explore plasma cytokines in pediatric and adult cases of MMD. We hypothesized that quantitative and qualitative differences exist between adults and children, and primary idiopathic versus secondary syndromic cases of MMD.

Methods: We performed plasma cytokine analysis using 63-plex Luminex immunoassay on 227 adults, 90% with primary MMD, and 30 children aged <21 years, 80% with primary MMD, from a single center. Patients with a history of stroke within 1-month of plasma collection were excluded. We fitted a linear regression model using pooled data, with adjustment for age and sex, to assess for differences between primary and secondary patients with respect to each of 60 cytokines. We then conducted a stratified analysis by fitting a set of linear regression models for adults and children separately to examine potential interactions between etiology and age. Statistical significance was adjusted using Bonferroni method to account for multiple testing.

Results: Three cytokines differed significantly between primary and secondary cases of MMD after Bonferroni correction (p<0.00083), with secondary cases exhibiting higher levels of MCP3 (p=3.8x10 -8 ), IP10 (p=5.1x10 -4 ), and IL12P40 (p=6.4x10 -4 ). Stratified analysis by age identified two cytokines, VEGF (p=7.2x10 -4 ) and IL12P70 (p=7.8x10 -4 ), that differed significantly among children, with higher levels among secondary versus primary MMD, while the difference was not significant among adults (interaction p<2x10 -4 ) (figure 1).

Conclusions: Syndromic cases of MMD demonstrate higher levels of plasma cytokines compared to primary cases of MMD, and the difference between primary versus secondary MMD is modified by age for some cytokines. This study suggests the importance of including age and etiology of MMD in investigating biomarkers and disease pathogenesis.

Longer-term outcomes of darbepoetin alfa versus epoetin alfa in patients with ESRD initiating hemodialysis: a quasi-experimental cohort study

BACKGROUND

Adequately powered studies directly comparing hard clinical outcomes of darbepoetin alfa (DPO) versus epoetin alfa (EPO) in patients undergoing dialysis are lacking.

STUDY DESIGN

Observational, registry-based, retrospective cohort study; we mimicked a cluster-randomized trial by comparing mortality and cardiovascular events in US patients initiating hemodialysis therapy in facilities (almost) exclusively using DPO versus EPO.

SETTING & PARTICIPANTS

Nonchain US hemodialysis facilities; each facility switching from EPO to DPO (2003-2010) was matched for location, profit status, and facility type with one EPO facility. Patients subsequently initiating hemodialysis therapy in these facilities were assigned their facility-level exposure.

INTERVENTION

DPO versus EPO.

OUTCOMES

All-cause mortality, cardiovascular mortality; composite of cardiovascular death, nonfatal myocardial infarction (MI), and nonfatal stroke.

MEASUREMENTS

Unadjusted and adjusted HRs from Cox proportional hazards regression models.

RESULTS

Of 508 dialysis facilities that switched to DPO, 492 were matched with a similar EPO facility; 19,932 (DPO: 9,465 [47.5%]; EPO: 10,467 [52.5%]) incident hemodialysis patients were followed up for 21,918 person-years during which 5,550 deaths occurred. Almost all baseline characteristics were tightly balanced. The demographics-adjusted mortality HR for DPO (vs EPO) was 1.06 (95% CI, 1.00-1.13) and was materially unchanged after adjustment for all other baseline characteristics (HR, 1.05; 95% CI, 0.99-1.12). Cardiovascular mortality did not differ between groups (HR, 1.05; 95% CI, 0.94-1.16). Nonfatal outcomes were evaluated among 9,455 patients with fee-for-service Medicare: 4,542 (48.0%) in DPO and 4,913 (52.0%) in EPO facilities. During 10,457 and 10,363 person-years, 248 and 372 events were recorded, respectively, for strokes and MIs. We found no differences in adjusted stroke or MI rates or their composite with cardiovascular death (HR, 1.10; 95% CI, 0.96-1.25).

LIMITATIONS

Nonrandom treatment assignment, potential residual confounding.

CONCLUSIONS

In incident hemodialysis patients, mortality and cardiovascular event rates did not differ between patients treated at facilities predominantly using DPO versus EPO.

Do changes in circulating biomarkers track with each other and with functional changes in older adults?

BACKGROUND

It is unclear if changes in proposed circulating biomarkers of aging are strongly correlated to each other or functional change. We tested if biomarker changes track with each other and with functional measures over 9 years in older adults.

METHODS

Dehydroepiandrosterone sulfate (DHEAS), adiponectin, insulin-like growth factor 1 (IGF-1), IGF binding proteins 1 (IGFBP-1) and 3 (IGFBP-3), interleukin-6 (IL-6), cholesterol, and function (gait speed, grip strength, Modified Mini Mental Status Exam [3MSE] and Digit Symbol Substitution Test [DSST] scores) were measured in 1996-1997 and 2005-2006 in the Cardiovascular Health Study All Stars study (N = 901, mean [standard deviation, SD] age 85.3 [3.6] years in 2005-2006). Adjusted Pearson correlations illustrated if biomarkers tracked together. Multivariable linear regression demonstrated if biomarker changes tracked with functional changes.

RESULTS

Correlations among biomarker changes were mostly <0.2. In models with each biomarker entered separately, a 1-SD increase biomarker change was associated with change in function as follows: grip strength (DHEAS β = 0.61kg, p = .001; IL-6 β = -0.46kg, p = .012; cholesterol men β = 0.79kg, p = .016); gait speed (DHEAS β = 0.02 meters per second, p = .039; IL-6 β = -0.018 meters per second, p = .049); and DSST score (DHEAS women β = 1.46, p = .004; IL-6 β = -0.83, p = .027). When biomarkers were entered in the same model, significant associations remaining were as follows: grip strength (DHEAS β = 0.54kg, p = .005; IL-6 β = -0.43kg, p = .022); 3MSE score (IGF-1 β = 0.96, p = .04; IGFBP-3 β = -1.07, p = .024); and DSST score (DHEAS women β = 1.27, p = .012; IL-6 β = -0.80, p = .04).

CONCLUSION

Changes in biomarkers were poorly correlated, supporting a model of stochastic, independent change across systems. DHEAS and IL-6 tracked most closely with function, illustrating that changes in inflammation and sex steroids may play dominant roles in changes of these functional outcomes.

Early dropout from psychotherapy for depression with group- and network-model therapists

Administrative data were used to examine early dropout among 16,451 health plan members calling to request psychotherapy for depression. Compared to members referred to group-model therapists, those referred to network-model therapists were more likely to drop out before the initial visit (OR 2.33, 95% CI 2.17-2.50) but less likely to drop out after the first visit (OR 0.45, 95% CI 0.43-0.48). These differences were unaffected by adjustment for neighborhood income and educational attainment, antidepressant use, or generosity of insurance coverage. Efforts to increase the effectiveness of psychotherapy may required different strategies in group- and network-model practice.

Impact of deductibles on initiation and continuation of psychotherapy for treatment of depression

OBJECTIVE

To estimate the impact of deductibles on the initiation and continuation of psychotherapy for depression.

DATA SOURCES/STUDY SETTING

Data from health care encounters and claims from Group Health Cooperative, a large integrated health care system in Washington State, was merged with information from a centralized behavioral health triage call center to conduct study analyses.

STUDY DESIGN

A retrospective observational design using a hierarchical logistic regression model was used to estimate initiation and continuation probabilities for use of psychotherapy, adjusting for key sociodemographic/economic factors and prior use of behavioral health services relevant to individual decisions to seek mental health care.

DATA COLLECTION/EXTRACTION METHODS

Analyses were based on merged datasets on patient enrollment, insurance benefits, use of mental health and general medical services and information collected by a triage specialist at a centralized behavioral health call center.

PRINCIPAL FINDINGS

Among individuals with unmet deductibles between $100 and $500, we found a statistically significant lower likelihood of making an initial visit, but there was no statistically significant effect on making an initial or subsequent visit among individuals that had met their deductible.

CONCLUSIONS

Unmet deductibles appear to influence the likelihood of initiating psychotherapy for treating depression.

Two new covariate adjustment methods for non-inferiority assessment of binary clinical trials data

In clinical trials, examining the adjusted treatment difference has become the preferred way to establish non-inferiority (NI) in cases involving a binary endpoint. However, current methods are inadequate in the area of covariate adjustment. In this paper, we introduce two new methods, nonparametric and parametric, of using the probability and probability (P-P) curve to address the issue of unadjusted categorical covariates in the traditional assessment of NI in clinical trials. We also show that the area under the P-P curve is a valid alternative for assessing NI using the adjusted treatment difference, and we compute this area using Mann-Whitney nonparametric statistics. Our simulation studies demonstrate that our proposed methods can not only control type I error at a predefined significance level but also achieve higher statistical power than those of traditional parametric and nonparametric methods that overlook covariate adjustment, especially when covariates are unbalanced in the two treatment groups. We illustrate the effectiveness of our methodology with data from clinical trials of a therapy for coronary heart disease.

Activation of insulin signal transduction pathway and anti-diabetic activity of small molecule insulin receptor activators

We recently described the identification of a non-peptidyl fungal metabolite (l-783,281, compound 1), which induced activation of human insulin receptor (IR) tyrosine kinase and mediated insulin-like effects in cells, as well as decreased blood glucose levels in murine models of Type 2 diabetes (Zhang, B., Salituro, G., Szalkowski, D., Li, Z., Zhang, Y., Royo, I., Vilella, D., Diez, M. T. , Pelaez, F., Ruby, C., Kendall, R. L., Mao, X., Griffin, P., Calaycay, J., Zierath, J. R., Heck, J. V., Smith, R. G. & Moller, D. E. (1999) Science 284, 974-977). Here we report the characterization of an active analog (compound 2) with enhanced IR kinase activation potency and selectivity over related receptors (insulin-like growth factor I receptor, epidermal growth factor receptor, and platelet-derived growth factor receptor). The IR activators stimulated tyrosine kinase activity of partially purified native IR and recombinant IR tyrosine kinase domain. Administration of the IR activators to mice was associated with increased IR tyrosine kinase activity in liver. In vivo oral treatment with compound 2 resulted in significant glucose lowering in several rodent models of diabetes. In db/db mice, oral administration of compound 2 elicited significant correction of hyperglycemia. In a streptozotocin-induced diabetic mouse model, compound 2 potentiated the glucose-lowering effect of insulin. In normal rats, compound 2 improved oral glucose tolerance with significant reduction in insulin release following glucose challenge. A structurally related inactive analog (compound 3) was not effective on insulin receptor activation or glucose lowering in db/db mice. Thus, small molecule IR activators exert insulin mimetic and sensitizing effects in cells and in animal models of diabetes. These results have implications for the future development of new therapies for diabetes mellitus.

Discovery of a potent, highly selective, and orally efficacious small-molecule activator of the insulin receptor

A series of 3,6-diaryl-2,5-dihydroxybenzoquinones were synthesized and evaluated for their abilities to selectively activate human insulin receptor tyrosine kinase (IRTK). 2, 5-Dihydroxy-6-(1-methylindol-3-yl)-3-phenyl-1,4-benzoquinone (2h) was identified as a potent, highly selective, and orally active small-molecule insulin receptor activator. It activated IRTK with an EC(50) of 300 nM and did not induce the activation of closely related receptors (IGFIR, EGFR, and PDGFR) at concentrations up to 30 000 nM. Oral administration of the compound to hyperglycemic db/db mice (0.1-10 mg/kg/day) elicited substantial to nearly complete correction of hyperglycemia in a dose-dependent manner. In ob/ob mice, the compound (10 mg/kg) caused significant reduction in hyperinsulinemia. A structurally related compound 2c, inactive in IRTK assay, failed to affect blood glucose level in db/db mice at equivalent exposure levels. Results from additional studies with compound 2h, aimed at evaluating classical quinone-related phenomena, provided sufficient grounds for optimism to allow more extensive toxicologic evaluation.