OWL: an optimized and independently validated machine learning prediction model for lung cancer screening based on the UK Biobank, PLCO, and NLST populations

Using a Polygenic Risk Score to Improve the Risk Prediction of Non-Small Cell Lung Cancer in Taiwan

PURPOSE

Low-dose computed tomography (LDCT) can help reducing lung cancer mortality. In Taiwan, the existing screening criteria revolve around smoking habits and family history of lung cancer. The role of genetic variation in non-small cell lung cancer (NSCLC) development is increasingly recognized. In this study, we aimed to investigate the potential benefits of polygenic risk scores (PRSs) in predicting NSCLC and enhancing the effectiveness of screening programs.

METHODS

We conducted a retrospective cohort study that included participants without prior diagnosis of lung cancer and later received LDCT for lung cancer screening. Genetic data for these participants were obtained from the project of Taiwan Precision Medicine Initiative. We adopted the model of genome-wide association study-derived PRS calculation using 19 susceptibility loci associated with the risk of NSCLC as reported by Dai et al.

RESULTS

We studied a total of 2,287 participants (1,197 male, 1,090 female). More female participants developed NSCLC during the follow-up period (4.4% v 2.5%, P = .015). The only risk factor of NSCLC diagnosis among male participants was age. Among female participants, independent risk factors of NSCLC diagnosis were age (adjusted hazard ratio [aHR], 1.08 [95% CI, 1.04 to 1.11]), a family history of lung cancer (aHR, 3.21 [95% CI, 1.78 to 5.77]), and PRS fourth quartile (aHR, 2.97 [95% CI, 1.25 to 7.07]). We used the receiver operating characteristics to show an AUC value of 0.741 for the conventional model. With the further incorporation of PRS, the AUC rose to 0.778.

CONCLUSION

The evaluation of PRS for NSCLC prediction holds promise for enhancing the effectiveness of lung cancer screening in Taiwan especially in women. By incorporating genetic information, screening criteria can be tailored to identify individuals at higher risks of NSCLC.

Evaluation of risk prediction models to select lung cancer screening participants in Europe: a prospective cohort consortium analysis

BACKGROUND

Lung cancer risk prediction models might efficiently identify individuals who should be offered lung cancer screening. However, their performance has not been comprehensively evaluated in Europe. We aimed to externally validate and evaluate the performance of several risk prediction models that predict lung cancer incidence or mortality in prospective European cohorts.

METHODS

We analysed 240 137 participants aged 45-80 years with a current or former smoking history from nine European countries in four prospective cohorts from the pooled database of the Lung Cancer Cohort Consortium: the Alpha-Tocopherol, Beta-Carotene Cancer Prevention Study (Finland), the Nord-Trøndelag Health Study (Norway), CONSTANCES (France), and the European Prospective Investigation into Cancer and Nutrition (Denmark, Germany, Italy, Spain, Sweden, the Netherlands, and Norway). We evaluated ten lung cancer risk models, which comprised the Bach, the Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial 2012 model (PLCOm2012), the Lung Cancer Risk Assessment Tool (LCRAT), the Lung Cancer Death Risk Assessment Tool (LCDRAT), the Nord-Trøndelag Health Study (HUNT), the Optimized Early Warning Model for Lung Cancer Risk (OWL), the University College London-Death (UCLD), the University College London-Incidence (UCLI), the Liverpool Lung Project version 2 (LLP version 2), and the Liverpool Lung Project version 3 (LLP version 3) models. We quantified model calibration as the ratio of expected to observed cases or deaths and discrimination using the area under the receiver operating characteristic curve (AUC). For each model, we also identified risk thresholds that would screen the same number of individuals as each of the US Preventive Services Task Force 2021 (USPSTF-2021), the US Preventive Services Task Force 2013 (USPSTF-2013), and the Nederlands-Leuvens Longkanker Screenings Onderzoek (NELSON) criteria.

FINDINGS

Among the participants, 1734 lung cancer cases and 1072 lung cancer deaths occurred within five years of enrolment. Most models had reasonable calibration in most countries, although the LLP version 2 overpredicted risk by more than 50% in eight countries (expected to observed ≥1·50). The PLCOm2012, LCDRAT, LCRAT, Bach, HUNT, OWL, UCLD, and UCLI models showed similar discrimination in most countries, with AUCs ranging from 0·68 (95% CI 0·59-0·77) to 0·83 (0·78-0·89), whereas the LLP version 2 and LLP version 3 showed lower discrimination, with AUCs ranging from 0·64 (95% CI 0·57-0·72) to 0·78 (0·74-0·83). When pooling data from all countries (but excluding the HUNT cohort), 33·9% (73 313 of 216 387) of individuals were eligible by USPSTF-2021 criteria, which included 74·8% (1185) of lung cancers and 76·3% (730) of lung cancer deaths occurring over 5 years. Fewer individuals were selected by USPSTF-2013 and NELSON criteria. After applying thresholds to select a population of equal size to USPSTF-2021, the PLCOm2012, LCDRAT, LCRAT, Bach, HUNT, OWL, UCLD, and UCLI, models identified 77·6%-79·1% of future cases, although they selected slightly older individuals compared with USPSTF-2021 criteria. Results were similar for USPSTF-2013 and NELSON.

INTERPRETATION

Several lung cancer risk prediction models showed good performance in European countries and might improve the efficiency of lung cancer screening if used in place of categorical eligibility criteria.

FUNDING

US National Cancer Institute, l'Institut National du Cancer, Cancer Research UK.

Radiomics based on multiple machine learning methods for diagnosing early bone metastases not visible on CT images

OBJECTIVES

This study utilizes [99mTc]-methylene diphosphate (MDP) single photon emission computed tomography (SPECT) images as a reference standard to evaluate whether the integration of radiomics features from computed tomography (CT) and machine learning algorithms can identify microscopic early bone metastases. Additionally, we also determine the optimal machine learning approach.

MATERIALS AND METHODS

We retrospectively studied 63 patients with early bone metastasis from July 2020 to March 2023. The ITK-SNAP software was used to delineate early bone metastases and normal bone tissue in SPECT images of each patient, which were then registered onto CT images to outline the volume of interest (VOI). The VOI includes 63 early bone metastasis volumes and 63 normal bone tissue volumes. 126 VOIs were randomly distributed in a 7:3 ratio between the training and testing groups, and 944 radiomics features were extracted from every VOI. We established 20 machine learning models using 5 feature selection algorithms and 4 classification methods. Evaluate the performance of the model using the area under the receiver operating characteristic curve (AUC).

RESULTS

Most machine learning models demonstrated outstanding discriminative capacity, with AUCs higher than 0.70. Notably, the K-Nearest Neighbors (KNN) classifier exhibited significant performance improvement compared to the other four classifiers. Specifically, the model constructed utilizing eXtreme Gradient Boosting (XGBoost) feature selection method integrated with KNN classifier achieved the maximum AUC, which is 0.989 in the training set and 0.975 in the testing set.

CONCLUSIONS

Radiomics features integrated with machine learning methods can identify early bone metastases that are not visible on CT images. In our analysis, KNN is considered the optimal classification method.

Use of ROC curve analysis for prediction gives fallacious results: Use predictivity-based indices

ABSTRACT

The area under the ROC curve is frequently used for assessing the predictive efficacy of a model, and the Youden index is commonly used to provide the optimal cut-off. Both are misleading tools for predictions. A ROC curve is drawn for the sensitivity of a quantitative test against its (1 - specificity) at different values of the test. Both sensitivity and specificity are retrospective in nature as these are indicators of correct classification of already known conditions. They are not indicators of future events and are not valid for predictions. Predictivity intimately depends on the prevalence which may be ignored by sensitivity and specificity. We explain this fallacy in detail and illustrate with several examples that the actual predictivity could differ greatly from the ROC curve-based predictivity reported by many authors. The predictive efficacy of a test or a model is best assessed by the percentage correctly predicted in a prospective framework. We propose predictivity-based ROC curves as tools for providing predictivities at varying prevalence in different populations. For optimal cut-off for prediction, in place of the Youden index, we propose a P-index where the sum of positive and negative predictivities is maximum after subtracting 1. To conclude, for correctly assessing adequacy of a prediction models, predictivity-based ROC curves should be used instead of the usual sensitivity-specificity-based ROC curves and the P-index should replace the Youden index.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

AI Deployment on GBM Diagnosis: A Novel Approach to Analyze Histopathological Images Using Image Feature-Based Analysis

BACKGROUND

Glioblastoma (GBM) is one of the most common malignant primary brain tumors, which accounts for 60-70% of all gliomas. Conventional diagnosis and the decision of post-operation treatment plan for glioblastoma is mainly based on the feature-based qualitative analysis of hematoxylin and eosin-stained (H&E) histopathological slides by both an experienced medical technologist and a pathologist. The recent development of digital whole slide scanners makes AI-based histopathological image analysis feasible and helps to diagnose cancer by accurately counting cell types and/or quantitative analysis. However, the technology available for digital slide image analysis is still very limited. This study aimed to build an image feature-based computer model using histopathology whole slide images to differentiate patients with glioblastoma (GBM) from healthy control (HC).

METHOD

Two independent cohorts of patients were used. The first cohort was composed of 262 GBM patients of the Cancer Genome Atlas Glioblastoma Multiform Collection (TCGA-GBM) dataset from the cancer imaging archive (TCIA) database. The second cohort was composed of 60 GBM patients collected from a local hospital. Also, a group of 60 participants with no known brain disease were collected. All the H&E slides were collected. Thirty-three image features (22 GLCM and 11 GLRLM) were retrieved from the tumor volume delineated by medical technologist on H&E slides. Five machine-learning algorithms including decision-tree (DT), extreme-boost (EB), support vector machine (SVM), random forest (RF), and linear model (LM) were used to build five models using the image features extracted from the first cohort of patients. Models built were deployed using the selected key image features for GBM diagnosis from the second cohort (local patients) as model testing, to identify and verify key image features for GBM diagnosis.

RESULTS

All five machine learning algorithms demonstrated excellent performance in GBM diagnosis and achieved an overall accuracy of 100% in the training and validation stage. A total of 12 GLCM and 3 GLRLM image features were identified and they showed a significant difference between the normal and the GBM image. However, only the SVM model maintained its excellent performance in the deployment of the models using the independent local cohort, with an accuracy of 93.5%, sensitivity of 86.95%, and specificity of 99.73%.

CONCLUSION

In this study, we have identified 12 GLCM and 3 GLRLM image features which can aid the GBM diagnosis. Among the five models built, the SVM model proposed in this study demonstrated excellent accuracy with very good sensitivity and specificity. It could potentially be used for GBM diagnosis and future clinical application.

See Lung Cancer with an AI

A lot has happened in the field of lung cancer screening in recent months. The ongoing discussion and documentation published by the scientific community and policymakers are of great importance to the entire European community and perhaps beyond. Lung cancer is the main worldwide killer. Low-dose computed tomography-based screening, together with smoking cessation, is the only tool to fight lung cancer, as it has already been proven in the United States of America but also European randomized controlled trials. Screening requires a lot of well-organized specialized work, but it can be supported by artificial intelligence (AI). Here we discuss whether and how to use AI for patients, radiologists, pulmonologists, thoracic surgeons, and all hospital staff supporting screening process benefits.