ROC Curve Analysis in the Presence of Imperfect Reference Standards

Evaluating the predictive performance of gut microbiota for the early-stage colorectal cancer

BACKGROUND

Colorectal cancer (CRC) has been regarded as one of the most frequently diagnosed malignancies among the leading causes of cancer-related morbidity and mortality globally. Diagnosis of CRC at the early-stages of tumour might improve the survival rate of patients. The current study sought to determine the performance of fecal Fusobacterium nucleatum (F. nucleatum) and Streptococcus bovis (S. bovis) for timely predicting CRC.

METHODS

Through a case-control study, the fecal sample information of 83 individuals (38 females, 45 males) referring to a hospital in Tehran, Iran was used. All patients underwent a complete colonoscopy, regarded as a gold standard test. Bacterial species including S. bovis and F. nucleatum were measured by absolute quantitative real-time PCR. The Bayesian univariate and bivariate latent class models (LCMs) were applied to estimate the ability of the candidate bacterial markers in order to early detection of patients with CRC.

RESULTS

Bayesian univariate LCMs demonstrated that the sensitivities of S. bovis and F. nucleatum were estimated to be 86% [95% credible interval (CrI) 0.82-0.91] and 82% (95% CrI 0.75-0.88); while specificities were 84% (95% CrI 0.78-0.89) and 80% (95% CrI 0.73-0.87), respectively. Moreover, the area under the receiver operating characteristic curves (AUCs) were 0.88 (95% CrI 0.83-0.94) and 0.80 (95% CrI 0.73-0.85) respectively for S. bovis and F. nucleatum. Based on the Bayesian bivariate LCMs, the sensitivities of S. bovis and F. nucleatum were calculated as 93% (95% CrI 0.84-0.98) and 90% (95% CrI 0.85-0.97), the specificities were 88% (95% CrI 0.78-0.93) and 87% (95% CrI 0.79-0.94); and the AUCs were 0.91 (95% CrI 0.83-0.99) and 0.88(95% CrI 0.81-0.96), respectively.

CONCLUSIONS

Our data has identified that according to the Bayesian bivariate LCM, S. bovis and F. nucleatum had a more significant predictive accuracy compared with the univariate model. In summary, these intestinal bacteria have been highlighted as novel tools for early-stage CRC diagnosis.

Performance of 3 Sets of Criteria for Potentially Inappropriate Prescribing in Older People to Identify Inadequate Drug Treatment

IMPORTANCE

Potentially inappropriate medications (PIMs) and potential prescribing omissions (PPOs) are used in research to reflect the quality of drug treatment in older people and have been suggested for inclusion in core outcome sets for evaluation of interventions for improved prescribing. Their validation so far, however, is primarily restricted to expert opinion-based processes.

OBJECTIVE

To evaluate the performance of 3 explicit PIM/PPO criteria sets as diagnostic tools to identify inadequate drug treatment in older patients.

DESIGN, SETTING, AND PARTICIPANTS

This diagnostic study analyzed patients aged 65 years or older consecutively included from 2 primary health care centers from October to November 2017. Data were analyzed from February to August 2022.

EXPOSURES

The PIMs/PPOs were concordantly identified by 2 specialist physicians (2018-2019) retrospectively after a planned physician visit, using 3 European PIM/PPO criteria sets and without knowledge of this diagnostic study.

MAIN OUTCOMES AND MEASURES

Area under the receiver operating characteristic (ROC) curve, reflecting the ability of PIM/PPO criteria sets to identify the reference standard of inadequate drug treatment, determined by 2 specialist physicians in consensus. Inadequate drug treatment implied that additional action related to the medication could be medically justified before the next regular visit.

RESULTS

A total of 302 patients were analyzed (median age, 74 [IQR, 69-81] years; 178 women [59%]; median number of drugs in the medication list, 6 [IQR, 3-9]); 98 patients (32%) had inadequate drug treatment. A total of 0 to 8 PIMs/PPOs per patient were identified using the Screening Tool of Older Persons' Prescriptions (STOPP)/Screening Tool to Alert to Right Treatment (START) criteria, 0 to 6 with the European EU(7)-PIM list, and 0 to 12 with the Swedish set of indicators of prescribing quality. The areas under the ROC curve for the 3 sets to identify the reference standard for inadequate drug treatment were 0.60 (95% CI, 0.53-0.66) for the STOPP/START criteria, 0.69 (95% CI, 0.63-0.75) for the EU(7)-PIM list, and 0.73 (95% CI, 0.67-0.80) for the Swedish set. For comparison, the area under the ROC curve was 0.71 (95% CI, 0.65-0.78) using the number of drugs in the medication list.

CONCLUSIONS AND RELEVANCE

In this diagnostic study, the evaluated PIM/PPO sets had poor to fair performance as diagnostic tools to identify inadequate drug treatment, comparable with a simple count of the number of drugs in the medication list. These findings suggest that use of PIMs/PPOs as indicators of drug treatment quality in core outcome sets for the evaluation of interventions for improved prescribing may need reconsideration.

Multiple Machine Learning Approaches Based on Postoperative Prediction of Pulmonary Complications in Patients With Emergency Cerebral Hemorrhage Surgery

Objective

This study aimed to create a prediction model of postoperative pulmonary complications for the patients with emergency cerebral hemorrhage surgery.

Methods

Patients with hemorrhage surgery who underwent cerebral hemorrhage surgery were included and divided into two groups: patients with or without pulmonary complications. Patient characteristics, previous history, laboratory tests, and interventions were collected. Univariate and multivariate logistic regressions were used to predict postoperative pulmonary infection. Multiple machine learning approaches have been used to compare their importance in predicting factors, namely K-nearest neighbor (KNN), stochastic gradient descent (SGD), support vector classification (SVC), random forest (RF), and logistics regression (LR), as they are the most successful and widely used models for clinical data.

Results

Three hundred and fifty four patients with emergency cerebral hemorrhage surgery between January 1, 2017 and December 31, 2020 were included in the study. 53.7% (190/354) of the patients developed postoperative pulmonary complications (PPC). Stepwise logistic regression analysis revealed four independent predictive factors associated with pulmonary complications, including current smoker, lymphocyte count, clotting time, and ASA score. In addition, the RF model had an ideal predictive performance.

Conclusions

According to our result, current smoker, lymphocyte count, clotting time, and ASA score were independent risks of pulmonary complications. Machine learning approaches can also provide more evidence in the prediction of pulmonary complications.

Machine Learning Prediction of Length of Stay in Adult Spinal Deformity Patients Undergoing Posterior Spine Fusion Surgery

(1) Background: Length of stay (LOS) is a commonly reported metric used to assess surgical success, patient outcomes, and economic impact. The focus of this study is to use a variety of machine learning algorithms to reliably predict whether a patient undergoing posterior spinal fusion surgery treatment for Adult Spine Deformity (ASD) will experience a prolonged LOS. (2) Methods: Patients undergoing treatment for ASD with posterior spinal fusion surgery were selected from the American College of Surgeon's NSQIP dataset. Prolonged LOS was defined as a LOS greater than or equal to 9 days. Data was analyzed with the Logistic Regression, Decision Tree, Random Forest, XGBoost, and Gradient Boosting functions in Python with the Sci-Kit learn package. Prediction accuracy and area under the curve (AUC) were calculated. (3) Results: 1281 posterior patients were analyzed. The five algorithms had prediction accuracies between 68% and 83% for posterior cases (AUC: 0.566-0.821). Multivariable regression indicated that increased Work Relative Value Units (RVU), elevated American Society of Anesthesiologists (ASA) class, and longer operating times were linked to longer LOS. (4) Conclusions: Machine learning algorithms can predict if patients will experience an increased LOS following ASD surgery. Therefore, medical resources can be more appropriately allocated towards patients who are at risk of prolonged LOS.

Deep learning for diagnosis and survival prediction in soft tissue sarcoma

BACKGROUND

Clinical management of soft tissue sarcoma (STS) is particularly challenging. Here, we used digital pathology and deep learning (DL) for diagnosis and prognosis prediction of STS.

PATIENTS AND METHODS

Our retrospective, multicenter study included a total of 506 histopathological slides from 291 patients with STS. The Cancer Genome Atlas cohort (240 patients) served as training and validation set. A second, multicenter cohort (51 patients) served as an additional test set. The use of the DL model (DLM) as a clinical decision support system was evaluated by nine pathologists with different levels of expertise. For prognosis prediction, 139 slides from 85 patients with leiomyosarcoma (LMS) were used. Area under the receiver operating characteristic (AUROC) and accuracy served as main outcome measures.

RESULTS

The DLM achieved a mean AUROC of 0.97 (±0.01) and an accuracy of 79.9% (±6.1%) in diagnosing the five most common STS subtypes. The DLM significantly improved the accuracy of the pathologists from 46.3% (±15.5%) to 87.1% (±11.1%). Furthermore, they were significantly faster and more certain in their diagnosis. In LMS, the mean AUROC in predicting the disease-specific survival status was 0.91 (±0.1) and the accuracy was 88.9% (±9.9%). Cox regression showed the DLM's prediction to be a significant independent prognostic factor (P = 0.008, hazard ratio 5.5, 95% confidence interval 1.56-19.7) in these patients, outperforming other risk factors.

CONCLUSIONS

DL can be used to accurately diagnose frequent subtypes of STS from conventional histopathological slides. It might be used for prognosis prediction in LMS, the most prevalent STS subtype in our cohort. It can also help pathologists to make faster and more accurate diagnoses. This could substantially improve the clinical management of STS patients.

Chromatin accessibility of circulating CD8+ T cells predicts treatment response to PD-1 blockade in patients with gastric cancer

Although tumor genomic profiling has identified small subsets of gastric cancer (GC) patients with clinical benefit from anti-PD-1 treatment, not all responses can be explained by tumor sequencing alone. We investigate epigenetic elements responsible for the differential response to anti-PD-1 therapy by quantitatively assessing the genome-wide chromatin accessibility of circulating CD8⁺ T cells in patients’ peripheral blood. Using an assay for transposase-accessible chromatin using sequencing (ATAC-seq), we identify unique open regions of chromatin that significantly distinguish anti-PD-1 therapy responders from non-responders. GC patients with high chromatin openness of circulating CD8⁺ T cells are significantly enriched in the responder group. Concordantly, patients with high chromatin openness at specific genomic positions of their circulating CD8⁺ T cells demonstrate significantly better survival than those with closed chromatin. Here we reveal that epigenetic characteristics of baseline CD8⁺ T cells can be used to identify metastatic GC patients who may benefit from anti-PD-1 therapy.

Anti-PD-1 therapy could induce a durable response in patients with gastric cancer, however biomarkers to predict response to immunotherapy are generally lacking. Here the authors report that openness of chromatin in circulating CD8⁺ T cells predicts treatment outcome in patients with metastatic gastric cancer treated with pembrolizumab.

Random Forest-Based Prediction of Outcome and Mortality in Patients with Traumatic Brain Injury Undergoing Primary Decompressive Craniectomy

BACKGROUND

Various prognostic models are used to predict mortality and functional outcome in patients after traumatic brain injury with a trend to incorporate machine learning protocols. None of these models is focused exactly on the subgroup of patients indicated for decompressive craniectomy. Evidence regarding efficiency of this surgery is still incomplete, especially in patients undergoing primary decompressive craniectomy with evacuation of traumatic mass lesions.

METHODS

In a prospective study with a 6-month follow-up period, we assessed postoperative outcome and mortality of 40 patients who underwent primary decompressive craniectomy for traumatic brain injuries during 2018-2019. The results were analyzed in relation to a wide spectrum of preoperatively available demographic, clinical, radiographic, and laboratory data. Random forest algorithms were trained for prediction of both mortality and unfavorable outcome, with their accuracy quantified by area under the receiver operating curves (AUCs) for out-of-bag samples.

RESULTS

At the end of the follow-up period, we observed mortality of 57.5%. Favorable outcome (Glasgow Outcome Scale [GOS] score 4-5) was achieved by 30% of our patients. Random forest-based prediction models constructed for 6-month mortality and outcome reached a moderate predictive ability, with AUC = 0.811 and AUC = 0.873, respectively. Random forest models trained on handpicked variables showed slightly decreased AUC = 0.787 for 6-month mortality and AUC = 0.846 for 6-month outcome and increased out-of-bag error rates.

CONCLUSIONS

Random forest algorithms show promising results in prediction of postoperative outcome and mortality in patients undergoing primary decompressive craniectomy. The best performance was achieved by Classification Random forest for 6-month outcome.

Predicting Postoperative Mortality After Metastatic Intraspinal Neoplasm Excision: Development of a Machine-Learning Approach

OBJECTIVE

Mortality following surgical resection of spinal tumors is a devastating outcome. Naïve Bayes machine learning algorithms may be leveraged in surgical planning to predict mortality. In this investigation, we use a Naïve Bayes classification algorithm to predict mortality following spinal tumor excision within 30 days of surgery.

METHODS

Patients who underwent laminectomies between 2006 and 2018 for excisions of intraspinal neoplasms were selected from the National Surgical Quality Initiative Program. Naïve Bayes classifier analysis was conducted in Python. The area under the receiver operating curve (AUC) was calculated to evaluate the classifier's ability to predict mortality within 30 days of surgery. Multivariable logistic regression analysis was performed in R to identify risk factors for 30-day postoperative mortality.

RESULTS

In total, 2094 spine tumor surgery patients were included in the study. The 30-day mortality rate was 5.16%. The classifier yielded an AUC of 0.898, which exceeds the predictive capacity of the National Surgical Quality Initiative Program mortality probability calculator's AUC of 0.722 (P < 0.0001). The multivariable regression indicated that smoking history, chronic obstructive pulmonary disease, disseminated cancer, bleeding disorder history, dyspnea, and low albumin levels were strongly associated with 30-day mortality.

CONCLUSIONS

The Naïve Bayes classifier may be used to predict 30-day mortality for patients undergoing spine tumor excisions, with an increasing degree of accuracy as the model better performs by learning continuously from the input patient data. Patient outcomes can be improved by identifying high-risk populations early using the algorithm and applying that data to inform preoperative decision making, as well as patient selection and education.

Profile of the SeptiCyte™ LAB gene expression assay to diagnose infection in critically ill patients

Sepsis is a severe and frequently occurring clinical syndrome, caused by the inflammatory response to infections. Recent studies on the human transcriptome during sepsis have yielded several gene-expression assays that might assist physicians during clinical assessment of patients suspected of sepsis. SeptiCyte™ LAB (Immunexpress, Seattle, WA) is the first gene expression assay that was cleared by the FDA in the United States to distinguish infectious from non-infectious causes of systemic inflammation in critically ill patients. The test consists of the simultaneous amplification of four RNA transcripts (CEACAM4, LAMP1, PLAC8, and PLA2G7) in whole blood using a quantitative real-time PCR reaction. This review provides an overview of the challenges in the diagnosis of sepsis, the development of gene expression signatures, and a detailed description of available clinical performance studies evaluating SeptiCyte™ LAB.

Differential diagnosis between benign and malignant pleural effusion with dual-energy spectral CT

PURPOSE

To investigate the value of spectral CT in the differential diagnosis of benign from malignant pleural effusion.

METHOD AND MATERIALS

14 patients with benign pleural effusion and 15 patients with malignant pleural effusion underwent non-contrast spectral CT imaging. These patients were later verified by the combination of disease history, clinical signs and other information with the consensus of surgeons and radiologists. Various Spectral CT image parameters measured for the effusion were as follows: CT numbers of the polychromatic 140kVp images, monochromatic images at 40keV and 100keV, the material density contents from the water, fat and blood-based material decomposition images, the effective atomic number and the spectral curve slope. These values were statistically compared with t test and logistic regression analysis between benign and malignant pleural effusion.

RESULTS

The CT value of benign and malignant pleural effusion in the polychromatic 140kVp images showed no differences (12.61±3.39HU vs. 14.71±5.03HU) (P>0.05), however, they were statistically different on the monochromatic images at 40keV (43.15±3.79 vs. 39.42±2.60, p = 0.005) and 100keV (9.11±1.38 vs. 6.52±2.04, p<0.001). There was difference in the effective atomic number value between the benign (7.87±0.08) and malignant pleural effusion (7.90±0.02) (P = 0.02). Using 6.32HU as the threshold for CT value measurement at 100keV, one could obtain sensitivity of 100% and specificity of 66.7% with area-under-curve of 0.843 for differentiating benign from malignant effusion. In addition, age and disease history were potential confounding factors for differentiating malignant pleural effusion from benign, since the older age (61.13±12.51 year-old vs48.57±12.33 year-old) as well as longer disease history (70.00±49.28 day vs.28.36±21.64 day) were more easily to be found in the malignant pleural effusion group than those in the benign pleural effusion group. By combining above five factors, one could obtain sensitivity of 100% and specificity of 71.4% with area-under-curve of 0.933 for differentiating benign from malignant effusion.

CONCLUSION

The CT value measurement at both high and low energy levels and the effective atomic number obtained in a single spectral CT scan can assist the differential diagnosis of benign from malignant pleural effusion.Combining them with patient age and disease history can further improve diagnostic performance.

CLINICAL RELEVANCE/APPLICATION

Clinical findings and Spectral CT imaging can provide significant evidences about the nature of pleural effusion.