Predicting Response to Neoadjuvant Chemotherapy with PET Imaging Using Convolutional Neural Networks

A Role for FDG PET Radiomics in Personalized Medicine?

Radiomics describes the extraction of multiple features from medical images, including molecular imaging modalities, that with bioinformatic approaches, provide additional clinically relevant information that may be invisible to the human eye. This information may complement standard radiological interpretation with data that may better characterize a disease or that may provide predictive or prognostic information. Progressing from predefined image features, often describing heterogeneity of voxel intensities within a volume of interest, there is increasing use of machine learning to classify disease characteristics and deep learning methods based on artificial neural networks that can learn features without a priori definition and without the need for preprocessing of images. There have been advances in standardization and harmonization of methods to a level that should support multicenter studies. However, in this relatively early phase of research in the field, there are limited aspects that have been adopted into routine practice. Most of the reports in the molecular imaging field describe radiomic approaches in cancer using 18F-fluorodeoxyglucose positron emission tomography (18F-FDG-PET). In this review, we will describe radiomics in molecular imaging and summarize the pertinent literature in lung cancer where reports are most prevalent and mature.

Prognostic value of FDG-PET radiomics with machine learning in pancreatic cancer

Patients with pancreatic cancer have a poor prognosis, therefore identifying particular tumor characteristics associated with prognosis is important. This study aims to investigate the utility of radiomics with machine learning using 18F-fluorodeoxyglucose (FDG)-PET in patients with pancreatic cancer. We enrolled 161 patients with pancreatic cancer underwent pretreatment FDG-PET/CT. The area of the primary tumor was semi-automatically contoured with a threshold of 40% of the maximum standardized uptake value, and 42 PET features were extracted. To identify relevant PET parameters for predicting 1-year survival, Gini index was measured using random forest (RF) classifier. Twenty-three patients were censored within 1 year of follow-up, and the remaining 138 patients were used for the analysis. Among the PET parameters, 10 features showed statistical significance for predicting overall survival. Multivariate analysis using Cox HR regression revealed gray-level zone length matrix (GLZLM) gray-level non-uniformity (GLNU) as the only PET parameter showing statistical significance. In RF model, GLZLM GLNU was the most relevant factor for predicting 1-year survival, followed by total lesion glycolysis (TLG). The combination of GLZLM GLNU and TLG stratified patients into three groups according to risk of poor prognosis. Radiomics with machine learning using FDG-PET in patients with pancreatic cancer provided useful prognostic information.

Artificial intelligence-assisted esophageal cancer management: Now and future

Esophageal cancer poses diagnostic, therapeutic and economic burdens in high-risk regions. Artificial intelligence (AI) has been developed for diagnosis and outcome prediction using various features, including clinicopathologic, radiologic, and genetic variables, which can achieve inspiring results. One of the most recent tasks of AI is to use state-of-the-art deep learning technique to detect both early esophageal squamous cell carcinoma and esophageal adenocarcinoma in Barrett’s esophagus. In this review, we aim to provide a comprehensive overview of the ways in which AI may help physicians diagnose advanced cancer and make clinical decisions based on predicted outcomes, and combine the endoscopic images to detect precancerous lesions or early cancer. Pertinent studies conducted in recent two years have surged in numbers, with large datasets and external validation from multi-centers, and have partly achieved intriguing results of expert’s performance of AI in real time. Improved pre-trained computer-aided diagnosis algorithms in the future studies with larger training and external validation datasets, aiming at real-time video processing, are imperative to produce a diagnostic efficacy similar to or even superior to experienced endoscopists. Meanwhile, supervised randomized controlled trials in real clinical practice are highly essential for a solid conclusion, which meets patient-centered satisfaction. Notably, ethical and legal issues regarding the black-box nature of computer algorithms should be addressed, for both clinicians and regulators.

Dose-Distribution-Driven PET Image-Based Outcome Prediction (DDD-PIOP): A Deep Learning Study for Oropharyngeal Cancer IMRT Application

Purpose

To develop a deep learning-based AI agent, DDD-PIOP (Dose-Distribution-Driven PET Image Outcome Prediction), for predicting ¹⁸FDG-PET image outcomes of oropharyngeal cancer (OPC) in response to intensity-modulated radiation therapy (IMRT).

Methods

DDD-PIOP uses pre-radiotherapy ¹⁸FDG-PET/CT images and the planned spatial dose distribution as the inputs, and it predicts the ¹⁸FDG-PET image outcomes in response to the planned IMRT delivery. This AI agent centralizes a customized convolutional neural network (CNN) as a deep learning approach, and it incorporates a few designs to enhance prediction accuracy. 66 OPC patients who received IMRT treatment on a sequential boost regime (2 Gy/daily fraction) were studied for DDD-PIOP development. 61 patients were used for AI agent training/validation, and the remaining five were used as independent tests. To evaluate the developed AI agent’s performance, the predicted mean standardized uptake values (SUVs) of gross tumor volume (GTV) and clinical target volume (CTV) were compared with the ground truth values. Overall SUV distribution accuracy was evaluated by gamma test passing rates under different criteria.

Results

The developed DDD-PIOP successfully generated ¹⁸FDG-PET image outcome predictions for five test patients. The predicted mean SUV values of GTV/CTV were 3.50/1.41, which were close to the ground-truth values of 3.57/1.51. In 2D-based gamma tests, the average passing rate was 92.1% using 5%/10 mm criteria, which was improved to 95.9%/93.2% when focusing on GTV/CTV regions. 3D gamma test passing rates were 98.7% using 5%/10 mm criteria, and the corresponding GTV/CTV results were 99.8%/99.4%.

Conclusion

The reported results suggest that the developed AI agent DDD-PIOP successfully predicted ¹⁸FDG-PET image outcomes with high quantitative accuracy. The generated voxel-based image outcome predictions could be used for treatment planning optimization prior to radiation delivery for the best individual-based outcome.

Reinventing radiation therapy with machine learning and imaging bio-markers (radiomics): State-of-the-art, challenges and perspectives

Radiation therapy is a pivotal cancer treatment that has significantly progressed over the last decade due to numerous technological breakthroughs. Imaging is now playing a critical role on deployment of the clinical workflow, both for treatment planning and treatment delivery. Machine-learning analysis of predefined features extracted from medical images, i.e. radiomics, has emerged as a promising clinical tool for a wide range of clinical problems addressing drug development, clinical diagnosis, treatment selection and implementation as well as prognosis. Radiomics denotes a paradigm shift redefining medical images as a quantitative asset for data-driven precision medicine. The adoption of machine-learning in a clinical setting and in particular of radiomics features requires the selection of robust, representative and clinically interpretable biomarkers that are properly evaluated on a representative clinical data set. To be clinically relevant, radiomics must not only improve patients' management with great accuracy but also be reproducible and generalizable. Hence, this review explores the existing literature and exposes its potential technical caveats, such as the lack of quality control, standardization, sufficient sample size, type of data collection, and external validation. Based upon the analysis of 165 original research studies based on PET, CT-scan, and MRI, this review provides an overview of new concepts, and hypotheses generating findings that should be validated. In particular, it describes evolving research trends to enhance several clinical tasks such as prognostication, treatment planning, response assessment, prediction of recurrence/relapse, and prediction of toxicity. Perspectives regarding the implementation of an AI-based radiotherapy workflow are presented.

Improving PET Imaging Acquisition and Analysis With Machine Learning: A Narrative Review With Focus on Alzheimer's Disease and Oncology

Machine learning (ML) algorithms have found increasing utility in the medical imaging field and numerous applications in the analysis of digital biomarkers within positron emission tomography (PET) imaging have emerged. Interest in the use of artificial intelligence in PET imaging for the study of neurodegenerative diseases and oncology stems from the potential for such techniques to streamline decision support for physicians providing early and accurate diagnosis and allowing personalized treatment regimens. In this review, the use of ML to improve PET image acquisition and reconstruction is presented, along with an overview of its applications in the analysis of PET images for the study of Alzheimer's disease and oncology.

Radiomics for precision medicine: Current challenges, future prospects, and the proposal of a new framework

The advancement of artificial intelligence concurrent with the development of medical imaging techniques provided a unique opportunity to turn medical imaging from mostly qualitative, to further quantitative and mineable data that can be explored for the development of clinical decision support systems (cDSS). Radiomics, a method for the high throughput extraction of hand-crafted features from medical images, and deep learning -the data driven modeling techniques based on the principles of simplified brain neuron interactions, are the most researched quantitative imaging techniques. Many studies reported on the potential of such techniques in the context of cDSS. Such techniques could be highly appealing due to the reuse of existing data, automation of clinical workflows, minimal invasiveness, three-dimensional volumetric characterization, and the promise of high accuracy and reproducibility of results and cost-effectiveness. Nevertheless, there are several challenges that quantitative imaging techniques face, and need to be addressed before the translation to clinical use. These challenges include, but are not limited to, the explainability of the models, the reproducibility of the quantitative imaging features, and their sensitivity to variations in image acquisition and reconstruction parameters. In this narrative review, we report on the status of quantitative medical image analysis using radiomics and deep learning, the challenges the field is facing, propose a framework for robust radiomics analysis, and discuss future prospects.

Nuclear medicine radiomics in precision medicine: why we can't do without artificial intelligence

In recent years, radiomics, defined as the extraction of large amounts of quantitative features from medical images, has gained emerging interest. Radiomics consists of the extraction of handcrafted features combined with sophisticated statistical methods or machine learning algorithms for modelling, or deep learning algorithms that both learn features from raw data and perform modelling. These features have the potential to serve as non-invasive biomarkers for tumor characterization, prognostic stratification and response prediction, thereby contributing to precision medicine. However, especially in nuclear medicine, variable results are obtained when using radiomics for these purposes. Individual studies show promising results, but due to small numbers of patients per study and little standardization, it is difficult to compare and validate results on other datasets. This review describes the radiomic pipeline, its applications and the increasing role of artificial intelligence within the field. Furthermore, the challenges that need to be overcome to achieve clinical translation are discussed, so that, eventually, radiomics, combined with clinical data and other biomarkers, can contribute to precision medicine, by providing the right treatment to the right patient, with the right dose, at the right time.

Artificial intelligence radiogenomics for advancing precision and effectiveness in oncologic care (Review)

The new era of artificial intelligence (AI) has introduced revolutionary data-driven analysis paradigms that have led to significant advancements in information processing techniques in the context of clinical decision-support systems. These advances have created unprecedented momentum in computational medical imaging applications and have given rise to new precision medicine research areas. Radiogenomics is a novel research field focusing on establishing associations between radiological features and genomic or molecular expression in order to shed light on the underlying disease mechanisms and enhance diagnostic procedures towards personalized medicine. The aim of the current review was to elucidate recent advances in radiogenomics research, focusing on deep learning with emphasis on radiology and oncology applications. The main deep learning radiogenomics architectures, together with the clinical questions addressed, and the achieved genetic or molecular correlations are presented, while a performance comparison of the proposed methodologies is conducted. Finally, current limitations, potentially understudied topics and future research directions are discussed.

Multiplex bioimaging of single-cell spatial profiles for precision cancer diagnostics and therapeutics

Cancers exhibit functional and structural diversity in distinct patients. In this mass, normal and malignant cells create tumor microenvironment that is heterogeneous among patients. A residue from primary tumors leaks into the bloodstream as cell clusters and single cells, providing clues about disease progression and therapeutic response. The complexity of these hierarchical microenvironments needs to be elucidated. Although tumors comprise ample cell types, the standard clinical technique is still the histology that is limited to a single marker. Multiplexed imaging technologies open new directions in pathology. Spatially resolved proteomic, genomic, and metabolic profiles of human cancers are now possible at the single-cell level. This perspective discusses spatial bioimaging methods to decipher the cascade of microenvironments in solid and liquid biopsies. A unique synthesis of top-down and bottom-up analysis methods is presented. Spatial multi-omics profiles can be tailored to precision oncology through artificial intelligence. Data-driven patient profiling enables personalized medicine and beyond.

Radiomics with artificial intelligence: a practical guide for beginners

Radiomics is a relatively new word for the field of radiology, meaning the extraction of a high number of quantitative features from medical images. Artificial intelligence (AI) is broadly a set of advanced computational algorithms that basically learn the patterns in the data provided to make predictions on unseen data sets. Radiomics can be coupled with AI because of its better capability of handling a massive amount of data compared with the traditional statistical methods. Together, the primary purpose of these fields is to extract and analyze as much and meaningful hidden quantitative data as possible to be used in decision support. Nowadays, both radiomics and AI have been getting attention for their remarkable success in various radiological tasks, which has been met with anxiety by most of the radiologists due to the fear of replacement by intelligent machines. Considering ever-developing advances in computational power and availability of large data sets, the marriage of humans and machines in future clinical practice seems inevitable. Therefore, regardless of their feelings, the radiologists should be familiar with these concepts. Our goal in this paper was three-fold: first, to familiarize radiologists with the radiomics and AI; second, to encourage the radiologists to get involved in these ever-developing fields; and, third, to provide a set of recommendations for good practice in design and assessment of future works.

A convolutional neural network-based system to classify patients using FDG PET/CT examinations

BACKGROUND

As the number of PET/CT scanners increases and FDG PET/CT becomes a common imaging modality for oncology, the demands for automated detection systems on artificial intelligence (AI) to prevent human oversight and misdiagnosis are rapidly growing. We aimed to develop a convolutional neural network (CNN)-based system that can classify whole-body FDG PET as 1) benign, 2) malignant or 3) equivocal.

METHODS

This retrospective study investigated 3485 sequential patients with malignant or suspected malignant disease, who underwent whole-body FDG PET/CT at our institute. All the cases were classified into the 3 categories by a nuclear medicine physician. A residual network (ResNet)-based CNN architecture was built for classifying patients into the 3 categories. In addition, we performed a region-based analysis of CNN (head-and-neck, chest, abdomen, and pelvic region).

RESULTS

There were 1280 (37%), 1450 (42%), and 755 (22%) patients classified as benign, malignant and equivocal, respectively. In the patient-based analysis, CNN predicted benign, malignant and equivocal images with 99.4, 99.4, and 87.5% accuracy, respectively. In region-based analysis, the prediction was correct with the probability of 97.3% (head-and-neck), 96.6% (chest), 92.8% (abdomen) and 99.6% (pelvic region), respectively.

CONCLUSION

The CNN-based system reliably classified FDG PET images into 3 categories, indicating that it could be helpful for physicians as a double-checking system to prevent oversight and misdiagnosis.

What can artificial intelligence teach us about the molecular mechanisms underlying disease?

While molecular imaging with positron emission tomography or single-photon emission computed tomography already reports on tumour molecular mechanisms on a macroscopic scale, there is increasing evidence that there are multiple additional features within medical images that can further improve tumour characterization, treatment prediction and prognostication. Early reports have already revealed the power of radiomics to personalize and improve patient management and outcomes. What remains unclear is how these additional metrics relate to underlying molecular mechanisms of disease. Furthermore, the ability to deal with increasingly large amounts of data from medical images and beyond in a rapid, reproducible and transparent manner is essential for future clinical practice. Here, artificial intelligence (AI) may have an impact. AI encompasses a broad range of 'intelligent' functions performed by computers, including language processing, knowledge representation, problem solving and planning. While rule-based algorithms, e.g. computer-aided diagnosis, have been in use for medical imaging since the 1990s, the resurgent interest in AI is related to improvements in computing power and advances in machine learning (ML). In this review we consider why molecular and cellular processes are of interest and which processes have already been exposed to AI and ML methods as reported in the literature. Non-small-cell lung cancer is used as an exemplar and the focus of this review as the most common tumour type in which AI and ML approaches have been tested and to illustrate some of the concepts.

Towards clinical application of image mining: a systematic review on artificial intelligence and radiomics

PURPOSE

The aim of this systematic review was to analyse literature on artificial intelligence (AI) and radiomics, including all medical imaging modalities, for oncological and non-oncological applications, in order to assess how far the image mining research stands from routine medical application. To do this, we applied a trial phases classification inspired from the drug development process.

METHODS

Among the articles we considered for inclusion from PubMed were multimodality AI and radiomics investigations, with a validation analysis aimed at relevant clinical objectives. Quality assessment of selected papers was performed according to the QUADAS-2 criteria. We developed the phases classification criteria for image mining studies.

RESULTS

Overall 34,626 articles were retrieved, 300 were selected applying the inclusion/exclusion criteria, and 171 high-quality papers (QUADAS-2 ≥ 7) were identified and analysed. In 27/171 (16%), 141/171 (82%), and 3/171 (2%) studies the development of an AI-based algorithm, radiomics model, and a combined radiomics/AI approach, respectively, was described. A total of 26/27(96%) and 1/27 (4%) AI studies were classified as phase II and III, respectively. Consequently, 13/141 (9%), 10/141 (7%), 111/141 (79%), and 7/141 (5%) radiomics studies were classified as phase 0, I, II, and III, respectively. All three radiomics/AI studies were categorised as phase II trials.

CONCLUSIONS

The results of the studies are promising but still not mature enough for image mining tools to be implemented in the clinical setting and be widely used. The transfer learning from the well-known drug development process, with some specific adaptations to the image mining discipline could represent the most effective way for radiomics and AI algorithms to become the standard of care tools.

11C-methionine-PET for differentiating recurrent brain tumor from radiation necrosis: radiomics approach with random forest classifier

Differentiating recurrent brain tumor from radiation necrosis is often difficult. This study aims to investigate the efficacy of 11C-methionine (MET)-PET radiomics for distinguishing recurrent brain tumor from radiation necrosis, as compared with conventional tumor-to-normal cortex (T/N) ratio evaluation. We enrolled 41 patients with metastatic brain tumor or glioma treated using radiation therapy who underwent MET-PET. The area with a standardized uptake value > 1.3 times that of the normal brain cortex was contoured. Forty-two PET features were extracted and used in a random forest classifier and the diagnostic performance was evaluated using a 10-fold cross-validation scheme. Gini index was measured to identify relevant PET parameters for classification. The reference standard was surgical histopathological analysis or more than 6 months of follow-up with MRI. Forty-four lesions were used for the analysis. Thirty-three and 11 lesions were confirmed as recurrent brain tumor and radiation necrosis, respectively. Radiomics and T/N ratio evaluation showed sensitivities of 90.1% and 60.6%, and specificities of 93.9% and 72.7% with areas under the curve of 0.98 and 0.73, respectively. Gray level co-occurrence matrix dissimilarity was the most pertinent feature for diagnosis. MET-PET radiomics yielded excellent outcome for differentiating recurrent brain tumor from radiation necrosis, which outperformed T/N ratio evaluation.

MRI-derived radiomics: methodology and clinical applications in the field of pelvic oncology

Personalized medicine aims at offering optimized treatment options and improved survival for cancer patients based on individual variability. The success of precision medicine depends on robust biomarkers. Recently, the requirement for improved non-biologic biomarkers that reflect tumor biology has emerged and there has been a growing interest in the automatic extraction of quantitative features from medical images, denoted as radiomics. Radiomics as a methodological approach can be applied to any image and most studies have focused on PET, CT, ultrasound, and MRI. Here, we aim to present an overview of the radiomics workflow as well as the major challenges with special emphasis on the use of multiparametric MRI datasets. We then reviewed recent studies on radiomics in the field of pelvic oncology including prostate, cervical, and colorectal cancer.

[Prediction of chemotherapy response in primary osteosarcoma using the machine learning technique on radiomic data]

INTRODUCTION

Osteosarcoma is the most common malignant bone tumor before 25 years of age. Response to neoadjuvant chemotherapy determines continuation of treatment and is also a powerful prognostic factor. There are currently no reliable ways to evaluate it early. The aim is to develop a method to predict the chemotherapy response using radiomics from pre-treatment MRI.

METHODS

Clinical characteristics and MRI of patients treated for local or metastatic osteosarcoma were collected retrospectively in the Rhône-Alpes region, from 2007 to 2016. On initial MRI exams, each tumor was segmented by expert radiologist and 87 radiomic features were extracted automatically. Univariate analysis was performed to assess each feature's association with histological response following neoadjuvante chemotherapy. To distinguish good histological responder from poor, we built predictive models based on support vector machines. Their classification performance was assessed with the area under operating characteristic curve receiver (AUROC) from test data.

RESULTS

The analysis focused on the MRIs of 69 patients, 55.1% (38/69) of whom were good histological responders. The model obtained by support vector machines from initial MRI radiomic data had an AUROC of 0.98, a sensitivity of 100% (IC 95% [100%-100%]) and specificity of 86% (IC 95% [59.7%-111%]).

DISCUSSION

Radiomic based on MRI data would predict the chemotherapy response before treatment initiation, in patients treated for osteosarcoma.

Exploring Tumor Heterogeneity Using PET Imaging: The Big Picture

Personalized medicine represents a major goal in oncology. It has its underpinning in the identification of biomarkers with diagnostic, prognostic, or predictive values. Nowadays, the concept of biomarker no longer necessarily corresponds to biological characteristics measured ex vivo but includes complex physiological characteristics acquired by different technologies. Positron-emission-tomography (PET) imaging is an integral part of this approach by enabling the fine characterization of tumor heterogeneity in vivo in a non-invasive way. It can effectively be assessed by exploring the heterogeneous distribution and uptake of a tracer such as 18F-fluoro-deoxyglucose (FDG) or by using multiple radiopharmaceuticals, each providing different information. These two approaches represent two avenues of development for the research of new biomarkers in oncology. In this article, we review the existing evidence that the measurement of tumor heterogeneity with PET imaging provide essential information in clinical practice for treatment decision-making strategy, to better select patients with poor prognosis for more intensive therapy or those eligible for targeted therapy.

Deep Convolutional Neural Network-Based Positron Emission Tomography Analysis Predicts Esophageal Cancer Outcome

In esophageal cancer, few prediction tools can be confidently used in current clinical practice. We developed a deep convolutional neural network (CNN) with 798 positron emission tomography (PET) scans of esophageal squamous cell carcinoma and 309 PET scans of stage I lung cancer. In the first stage, we pretrained a 3D-CNN with all PET scans for a task to classify the scans into esophageal cancer or lung cancer. Overall, 548 of 798 PET scans of esophageal cancer patients were included in the second stage with an aim to classify patients who expired within or survived more than one year after diagnosis. The area under the receiver operating characteristic curve (AUC) was used to evaluate model performance. In the pretrain model, the deep CNN attained an AUC of 0.738 in identifying patients who expired within one year after diagnosis. In the survival analysis, patients who were predicted to be expired but were alive at one year after diagnosis had a 5-year survival rate of 32.6%, which was significantly worse than the 5-year survival rate of the patients who were predicted to survive and were alive at one year after diagnosis (50.5%, p < 0.001). These results suggest that the prediction model could identify tumors with more aggressive behavior. In the multivariable analysis, the prediction result remained an independent prognostic factor (hazard ratio: 2.830; 95% confidence interval: 2.252–3.555, p < 0.001). We conclude that a 3D-CNN can be trained with PET image datasets to predict esophageal cancer outcome with acceptable accuracy.

PET-detected pneumonitis following curative-intent chemoradiation in non-small cell lung cancer (NSCLC): recognizing patterns and assessing the impact on the predictive ability of FDG-PET/CT response assessment

PURPOSE

Inflammatory FDG uptake in the lung (PET-pneumonitis) following curative-intent radiotherapy (RT)/chemo-RT (CRT) in non-small cell lung cancer (NSCLC) can pose a challenge in FDG-PET/CT response assessment. The aim of this study is to describe different patterns of PET-pneumonitis to guide the interpretation of FDG-PET/CT and investigate its association with tumor response and overall survival (OS).

METHODS

Retrospective analysis was performed on 87 NSCLC patients in three prospective trials who were treated with radical RT (n = 7) or CRT (n = 80), with baseline and post-treatment FDG-PET/CT. Visual criteria were performed for post-treatment FDG-PET/CT response assessment. The grading of PET-pneumonitis was based on relative lung uptake intensity compared to organs of reference and classified as per Deauville score from grade 1-5. Distribution patterns of PET-pneumonitis were defined as follows: A) patchy/sub-pleural; B) diffuse (involving more than a segment); and C) peripheral (diffusely surrounding a photopenic region).

RESULTS

Follow-up FDG-PET/CT scans were performed approximately 3 months (median, 89 days; interquartile range, 79-93) after RT. Overall, PET-pneumonitis was present in 62/87 (71%) of patients, with Deauville 2 or 3 in 12/62 (19%) and 4 or 5 in 50/62 (81%) of patients. The frequency of patterns A, B and C of PET-pneumonitis was 19/62 (31%), 20/62 (32%) and 23/62 (37%), respectively. No association was found between grade or pattern of PET-pneumonitis and overall response at follow-up PET/CT (p = 0.27 and p = 0.56, respectively). There was also no significant association between PET-pneumonitis and OS (hazard ratio [HR], 1.3; 95% confidence interval [CI], 0.6-2.5; p = 0.45). Early FDG-PET/CT response assessment, however, was prognostic for OS (HR, 1.7; 95% CI, 1.2-2.2; p < 0.001).

CONCLUSION

PET-pneumonitis is common in early post-CRT/RT, but pattern recognition may assist in response assessment by FDG-PET/CT. While FDG-PET/CT is a powerful tool for response assessment and prognostication, PET-pneumonitis does not appear to confound early response assessment or to independently predict OS.

A convolutional neural network-based system to prevent patient misidentification in FDG-PET examinations

Patient misidentification in imaging examinations has become a serious problem in clinical settings. Such misidentification could be prevented if patient characteristics such as sex, age, and body weight could be predicted based on an image of the patient, with an alert issued when a mismatch between the predicted and actual patient characteristic is detected. Here, we tested a simple convolutional neural network (CNN)-based system that predicts patient sex from FDG PET-CT images. This retrospective study included 6,462 consecutive patients who underwent whole-body FDG PET-CT at our institute. The CNN system was used for classifying these patients by sex. Seventy percent of the randomly selected images were used to train and validate the system; the remaining 30% were used for testing. The training process was repeated five times to calculate the system's accuracy. When images for the testing were given to the learned CNN model, the sex of 99% of the patients was correctly categorized. We then performed an image-masking simulation to investigate the body parts that are significant for patient classification. The image-masking simulation indicated the pelvic region as the most important feature for classification. Finally, we showed that the system was also able to predict age and body weight. Our findings demonstrate that a CNN-based system would be effective to predict the sex of patients, with or without age and body weight prediction, and thereby prevent patient misidentification in clinical settings.

Machine-learned target volume delineation of 18F-FDG PET images after one cycle of induction chemotherapy

Biological tumour volume (GTV_PET) delineation on ¹⁸F-FDG PET acquired during induction chemotherapy (ICT) is challenging due to the reduced metabolic uptake and volume of the GTV_PET. Automatic segmentation algorithms applied to ¹⁸F-FDG PET (PET-AS) imaging have been used for GTV_PET delineation on ¹⁸F-FDG PET imaging acquired before ICT. However, their role has not been investigated in ¹⁸F-FDG PET imaging acquired after ICT. In this study we investigate PET-AS techniques, including ATLAAS a machine learned method, for accurate delineation of the GTV_PET after ICT. Twenty patients were enrolled onto a prospective phase I study (FiGaRO). PET/CT imaging was acquired at baseline and 3 weeks following 1 cycle of induction chemotherapy. The GTV_PET was manually delineated by a nuclear medicine physician and clinical oncologist. The resulting GTV_PET was used as the reference contour. The ATLAAS original statistical model was expanded to include images of reduced metabolic activity and the ATLAAS algorithm was re-trained on the new reference dataset. Estimated GTV_PET contours were derived using sixteen PET-AS methods and compared to the GTV_PET using the Dice Similarity Coefficient (DSC). The mean DSC for ATLAAS, 60% Peak Thresholding (PT60), Adaptive Thresholding (AT) and Watershed Thresholding (WT) was 0.72, 0.61, 0.63 and 0.60 respectively. The GTV_PET generated by ATLAAS compared favourably with manually delineated volumes and in comparison, to other PET-AS methods, was more accurate for GTV_PET delineation after ICT. ATLAAS would be a feasible method to reduce inter-observer variability in multi-centre trials.

Machine learning for prediction of chemoradiation therapy response in rectal cancer using pre-treatment and mid-radiation multi-parametric MRI

PURPOSE

To predict the neoadjuvant chemoradiation therapy (CRT) response in patients with locally advanced rectal cancer (LARC) using radiomics and deep learning based on pre-treatment MRI and a mid-radiation follow-up MRI taken 3-4 weeks after the start of CRT.

METHODS

A total of 51 patients were included, 45 with pre-treatment, 41 with mid-radiation therapy (RT), and 35 with both MRI sets. The multi-parametric MRI protocol included T2, diffusion weighted imaging (DWI) with b-values of 0 and 800 s/mm², and dynamic-contrast-enhanced (DCE) MRI. After completing CRT and surgery, the specimen was examined to determine the pathological response based on the tumor regression grade. The tumor ROI was manually drawn on the post-contrast image and mapped to other sequences. The total tumor volume and mean apparent diffusion coefficient (ADC) were measured. Radiomics using GLCM texture and histogram parameters, and deep learning using a convolutional neural network (CNN), were performed to differentiate pathologic complete response (pCR) vs. non-pCR, and good response (GR) vs. non-GR.

RESULTS

Tumor volume decreased and ADC increased significantly in the mid-RT MRI compared to the pre-treatment MRI. For predicting pCR vs. non-pCR, combining ROI and radiomics features achieved an AUC of 0.80 for pre-treatment, 0.82 for mid-RT, and 0.86 for both MRI together. For predicting GR vs. non-GR, the AUC was 0.91 for pre-treatment, 0.92 for mid-RT, and 0.93 for both MRI together. In deep learning using CNN, combining pre-treatment and mid-RT MRI achieved a higher accuracy compared to using either dataset alone, with AUC of 0.83 for predicting pCR vs. non-pCR.

CONCLUSION

Radiomics based on pre-treatment and early follow-up multi-parametric MRI in LARC patients receiving CRT could extract comprehensive quantitative information to predict final pathologic response.

Advanced PET imaging in oncology: status and developments with current and future relevance to lung cancer care

PURPOSE OF REVIEW

This review highlights the status and developments of PET imaging in oncology, with particular emphasis on lung cancer. We discuss the significance of PET for diagnosis, staging, decision-making, monitoring of treatment response, and drug development. The PET key advantage, the noninvasive assessment of functional and molecular tumor characteristics including tumor heterogeneity, as well as PET trends relevant to cancer care are exemplified.

RECENT FINDINGS

Advances of PET and radiotracer technology are encouraging for multiple fields of oncological research and clinical application, including in-depth assessment of PET images by texture analysis (radiomics). Whole body PET imaging and novel PET tracers allow assessing characteristics of most types of cancer. However, only few PET tracers in addition to F-fluorodeoxyglucose have sufficiently been validated, approved, and are reimbursed for a limited number of indications. Therefore, validation and standardization of PET parameters including tracer dosage, image acquisition, post processing, and reading are required to expand PET imaging as clinically applicable approach.

SUMMARY

Considering the potential of PET imaging for precision medicine and drug development in lung and other types of cancer, increasing efforts are warranted to standardize PET technology and to provide evidence for PET imaging as a guiding biomarker in nearly all areas of cancer treatment.

Image-based biomarkers for solid tumor quantification

The last few decades have witnessed tremendous technological developments in image-based biomarkers for tumor quantification and characterization. Initially limited to manual one- and two-dimensional size measurements, image biomarkers have evolved to harness developments not only in image acquisition technology but also in image processing and analysis algorithms. At the same time, clinical validation remains a major challenge for the vast majority of these novel techniques, and there is still a major gap between the latest technological developments and image biomarkers used in everyday clinical practice. Currently, the imaging biomarker field is attracting increasing attention not only because of the tremendous interest in cutting-edge therapeutic developments and personalized medicine but also because of the recent progress in the application of artificial intelligence (AI) algorithms to large-scale datasets. Thus, the goal of the present article is to review the current state of the art for image biomarkers and their use for characterization and predictive quantification of solid tumors. Beginning with an overview of validated imaging biomarkers in current clinical practice, we proceed to a review of AI-based methods for tumor characterization, such as radiomics-based approaches and deep learning.Key Points• Recent years have seen tremendous technological developments in image-based biomarkers for tumor quantification and characterization.• Image-based biomarkers can be used on an ongoing basis, in a non-invasive (or mildly invasive) way, to monitor the development and progression of the disease or its response to therapy.• We review the current state of the art for image biomarkers, as well as the recent developments in artificial intelligence (AI) algorithms for image processing and analysis.

Radiomics: Principles and radiotherapy applications

Radiomics is defined as the extraction of a large quantity of quantitative image features. The different radiomic indexes that have been proposed in the literature are described as well as the various factors that have an impact on the robustness of these indexes. We will see that several hundred quantitative features can be extracted per lesion and imaging modality. The ever-growing number of features studied raises the question of the statistical method of analysis used. This review addresses the research supporting the clinical use of radiomics in oncology in the staging of disease, discrimination between healthy and pathological tissues, the identification of genetic features, the prediction of patient survival, the response to treatment, the recurrence after radiotherapy and chemoradiotherapy and the side effects. Based on the existing literature, it remains difficult to identify features that should be used for current clinical practice.

Rice Blast Disease Recognition Using a Deep Convolutional Neural Network

Rice disease recognition is crucial in automated rice disease diagnosis systems. At present, deep convolutional neural network (CNN) is generally considered the state-of-the-art solution in image recognition. In this paper, we propose a novel rice blast recognition method based on CNN. A dataset of 2906 positive samples and 2902 negative samples is established for training and testing the CNN model. In addition, we conduct comparative experiments for qualitative and quantitatively analysis in our evaluation of the effectiveness of the proposed method. The evaluation results show that the high-level features extracted by CNN are more discriminative and effective than traditional hand-crafted features including local binary patterns histograms (LBPH) and Haar-WT (Wavelet Transform). Moreover, quantitative evaluation results indicate that CNN with Softmax and CNN with support vector machine (SVM) have similar performances, with higher accuracy, larger area under curve (AUC), and better receiver operating characteristic (ROC) curves than both LBPH plus an SVM as the classifier and Haar-WT plus an SVM as the classifier. Therefore, our CNN model is a top performing method for rice blast disease recognition and can be potentially employed in practical applications.

Convolutional Neural Networks for Radiologic Images: A Radiologist's Guide

Deep learning has rapidly advanced in various fields within the past few years and has recently gained particular attention in the radiology community. This article provides an introduction to deep learning technology and presents the stages that are entailed in the design process of deep learning radiology research. In addition, the article details the results of a survey of the application of deep learning-specifically, the application of convolutional neural networks-to radiologic imaging that was focused on the following five major system organs: chest, breast, brain, musculoskeletal system, and abdomen and pelvis. The survey of the studies is followed by a discussion about current challenges and future trends and their potential implications for radiology. This article may be used as a guide for radiologists planning research in the field of radiologic image analysis using convolutional neural networks.

Deep Learning: Current and Emerging Applications in Medicine and Technology

Machine learning is enabling researchers to analyze and understand increasingly complex physical and biological phenomena in traditional fields such as biology, medicine, and engineering and emerging fields like synthetic biology, automated chemical synthesis, and biomanufacturing. These fields require new paradigms toward understanding increasingly complex data and converting such data into medical products and services for patients. The move toward deep learning and complex modeling is an attempt to bridge the gap between acquiring massive quantities of complex data, and converting such data into practical insights. Here, we provide an overview of the field of machine learning, its current applications and needs in traditional and emerging fields, and discuss an illustrative attempt at using deep learning to understand swarm behavior of molecular shuttles.

Detection of pulmonary ground-glass opacity based on deep learning computer artificial intelligence

Background

A deep learning computer artificial intelligence system is helpful for early identification of ground glass opacities (GGOs).

Methods

Images from the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) database were used in AlexNet and GoogLeNet to detect pulmonary nodules, and 221 GGO images provided by Xinhua Hospital were used in ResNet50 for detecting GGOs. We used computed tomography image radial reorganization to create the input image of the three-dimensional features, and used the extracted features for deep learning, network training, testing, and analysis.

Results

In the final evaluation results, we found that the accuracy of identification of lung nodule could reach 88.0%, with an F-score of 0.891. In terms of performance and accuracy, our method was better than the existing solutions. The GGO nodule classification achieved the best F-score of 0.87805. We propose a preprocessing method of red, green, and blue (RGB) superposition in the region of interest to effectively increase the differentiation between nodules and normal tissues, and that is the innovation of our research.

Conclusions

The method of deep learning proposed in this study is more sensitive than other systems in recent years, and the average false positive is lower than that of others.

Radiogenomics for Precision Medicine With a Big Data Analytics Perspective

Precision medicine promises better healthcare delivery by improving clinical practice. Using evidence-based substratification of patients, the objective is to achieve better prognosis, diagnosis, and treatment that will transform existing clinical pathways toward optimizing care for the specific needs of each patient. The wealth of today's healthcare data, often characterized as big data, provides invaluable resources toward new knowledge discovery that has the potential to advance precision medicine. The latter requires interdisciplinary efforts that will capitalize the information, know-how, and medical data of newly formed groups fusing different backgrounds and expertise. The objective of this paper is to provide insights with respect to the state-of-the-art research in precision medicine. More specifically, our goal is to highlight the fundamental challenges in emerging fields of radiomics and radiogenomics by reviewing the case studies of Cancer and Alzheimer's disease, describe the computational challenges from a big data analytics perspective, and discuss standardization and open data initiatives that will facilitate the adoption of precision medicine methods and practices.

Quantitative imaging of cancer in the postgenomic era: Radio(geno)mics, deep learning, and habitats

Although cancer often is referred to as "a disease of the genes," it is indisputable that the (epi)genetic properties of individual cancer cells are highly variable, even within the same tumor. Hence, preexisting resistant clones will emerge and proliferate after therapeutic selection that targets sensitive clones. Herein, the authors propose that quantitative image analytics, known as "radiomics," can be used to quantify and characterize this heterogeneity. Virtually every patient with cancer is imaged radiologically. Radiomics is predicated on the beliefs that these images reflect underlying pathophysiologies, and that they can be converted into mineable data for improved diagnosis, prognosis, prediction, and therapy monitoring. In the last decade, the radiomics of cancer has grown from a few laboratories to a worldwide enterprise. During this growth, radiomics has established a convention, wherein a large set of annotated image features (1-2000 features) are extracted from segmented regions of interest and used to build classifier models to separate individual patients into their appropriate class (eg, indolent vs aggressive disease). An extension of this conventional radiomics is the application of "deep learning," wherein convolutional neural networks can be used to detect the most informative regions and features without human intervention. A further extension of radiomics involves automatically segmenting informative subregions ("habitats") within tumors, which can be linked to underlying tumor pathophysiology. The goal of the radiomics enterprise is to provide informed decision support for the practice of precision oncology.

Novel imaging techniques in staging oesophageal cancer

The survival of oesophageal cancer is poor as most patients present with advanced disease. Radiological staging of oesophageal cancer is complex but is fundamental to clinical management. Accurate staging investigations are vitally important to guide treatment decisions and optimise patient outcomes. A combination of baseline computed tomography (CT), endoscopic ultrasound (EUS) and positron emission tomography (PET) are currently used for initial treatment decisions. The potential value of these imaging modalities to re-stage disease, monitor response and alter treatment is currently being investigated. This review presents an essential update on the accuracy of oesophageal cancer staging investigations, their use in re-staging after neo-adjuvant therapy and introduces evolving imaging techniques, including novel biomarkers that have clinical potential in oesophageal cancer.

Challenges and Promises of PET Radiomics

PURPOSE

Radiomics describes the extraction of multiple, otherwise invisible, features from medical images that, with bioinformatic approaches, can be used to provide additional information that can predict underlying tumor biology and behavior.

METHODS AND MATERIALS

Radiomic signatures can be used alone or with other patient-specific data to improve tumor phenotyping, treatment response prediction, and prognosis, noninvasively. The data describing 18F-fluorodeoxyglucose positron emission tomography radiomics, often using texture or heterogeneity parameters, are increasing rapidly.

RESULTS

In relation to radiation therapy practice, early data have reported the use of radiomic approaches to better define tumor volumes and predict radiation toxicity and treatment response.

CONCLUSIONS

Although at an early stage of development, with many technical challenges remaining and a need for standardization, promise nevertheless exists that PET radiomics will contribute to personalized medicine, especially with the availability of increased computing power and the development of machine-learning approaches for imaging.

Applications of Deep Learning and Reinforcement Learning to Biological Data

Rapid advances in hardware-based technologies during the past decades have opened up new possibilities for life scientists to gather multimodal data in various application domains, such as omics, bioimaging, medical imaging, and (brain/body)-machine interfaces. These have generated novel opportunities for development of dedicated data-intensive machine learning techniques. In particular, recent research in deep learning (DL), reinforcement learning (RL), and their combination (deep RL) promise to revolutionize the future of artificial intelligence. The growth in computational power accompanied by faster and increased data storage, and declining computing costs have already allowed scientists in various fields to apply these techniques on data sets that were previously intractable owing to their size and complexity. This paper provides a comprehensive survey on the application of DL, RL, and deep RL techniques in mining biological data. In addition, we compare the performances of DL techniques when applied to different data sets across various application domains. Finally, we outline open issues in this challenging research area and discuss future development perspectives.

Positron emission tomography-based assessment of metabolic gradient and other prognostic features in sarcoma

Intratumoral heterogeneity biomarkers derived from positron emission tomography (PET) imaging with fluorodeoxyglucose (FDG) are of interest for a number of cancers, including sarcoma. A range of radiomic texture variables, adapted from general methodologies for image analysis, has shown promise in the setting. In the context of sarcoma, our group introduced an alternative model-based approach to the measurement of heterogeneity. In this approach, the heterogeneity of a tumor is characterized by the extent to which the 3-D FDG uptake pattern deviates from a simple elliptically contoured structure. By using a nonparametric analysis of the uptake profile obtained from this spatial model, a variable assessing the metabolic gradient of the tumor is developed. The work explores the prognostic potential of this new variable in the context of FDG-PET imaging of sarcoma. A mature clinical series involving 197 patients, 88 of whom have complete time-to-death information, is used. Texture variables based on the imaging data are also evaluated in this series and a range of appropriate machine learning methodologies are then used to explore the complementary prognostic roles for structure and texture variables. We conclude that both texture-based and model-based variables can be combined to achieve enhanced prognostic assessments of outcome for patients with sarcoma based on FDG-PET imaging information.

Increased evidence for the prognostic value of FDG uptake on late-treatment PET in non-tumour-affected oesophagus in irradiated patients with oesophageal carcinoma

PURPOSE

¹⁸F-FDG uptake in irradiated non-tumour-affected oesophagus (NTO) on restaging PET is a potential surrogate for the measurement of radiation-induced inflammation. Radiation-induced inflammation itself has been shown to be of high prognostic relevance in patients undergoing preoperative radiochemotherapy (RCT) for locally advanced oesophageal cancer. We assessed the prognostic relevance of FDG uptake in the NTO in an independent cohort of patients treated with definitive RCT.

METHODS

This retrospective evaluation included 72 patients with oesophageal squamous cell carcinoma treated with definitive RCT with curative intent. All patients underwent pretreatment and restaging FDG PET after receiving a radiation dose of 40-50 Gy. Standardized uptake values (SUV_max/SUV_mean), metabolic tumour volume (MTV) and relative changes from pretreatment to restaging PET (∆SUV_max/∆SUV_mean) were determined within the tumour and NTO. Univariate Cox regression with respect to overall survival (OS), local control (LC), distant metastases (DM) and treatment failure (TF) was performed. Independence of parameters was tested by multivariate Cox regression.

RESULTS

∆SUV_max NTO and MTV were prognostic factors for all investigated clinical endpoints (OS, LC, DM, TF). Inclusion of clinical and PET tumour parameters in multivariate analysis showed that ∆SUV_max NTO was an independent prognostic factor. Furthermore, multivariate analysis of ∆SUV_max NTO using previously published cut-off values from preoperatively treated patients revealed that ∆SUV_max NTO was independent prognostic factor for OS (HR = 1.88, p = 0.038), TF (HR = 2.11, p = 0.048) and DM (HR = 3.02, p = 0.047).

CONCLUSION

NTO-related tracer uptake during the course of treatment in patients with oesophageal carcinoma was shown to be of high prognostic relevance. Thus, metabolically activity of NTO measured in terms of ∆SUV_max NTO is a potential candidate for future treatment individualization (i.e. organ preservation).

Predicting Deletion of Chromosomal Arms 1p/19q in Low-Grade Gliomas from MR Images Using Machine Intelligence

Several studies have linked codeletion of chromosome arms 1p/19q in low-grade gliomas (LGG) with positive response to treatment and longer progression-free survival. Hence, predicting 1p/19q status is crucial for effective treatment planning of LGG. In this study, we predict the 1p/19q status from MR images using convolutional neural networks (CNN), which could be a non-invasive alternative to surgical biopsy and histopathological analysis. Our method consists of three main steps: image registration, tumor segmentation, and classification of 1p/19q status using CNN. We included a total of 159 LGG with 3 image slices each who had biopsy-proven 1p/19q status (57 non-deleted and 102 codeleted) and preoperative postcontrast-T1 (T1C) and T2 images. We divided our data into training, validation, and test sets. The training data was balanced for equal class probability and was then augmented with iterations of random translational shift, rotation, and horizontal and vertical flips to increase the size of the training set. We shuffled and augmented the training data to counter overfitting in each epoch. Finally, we evaluated several configurations of a multi-scale CNN architecture until training and validation accuracies became consistent. The results of the best performing configuration on the unseen test set were 93.3% (sensitivity), 82.22% (specificity), and 87.7% (accuracy). Multi-scale CNN with their self-learning capability provides promising results for predicting 1p/19q status non-invasively based on T1C and T2 images. Predicting 1p/19q status non-invasively from MR images would allow selecting effective treatment strategies for LGG patients without the need for surgical biopsy.

Towards precision medicine: from quantitative imaging to radiomics

Radiology (imaging) and imaging-guided interventions, which provide multi-parametric morphologic and functional information, are playing an increasingly significant role in precision medicine. Radiologists are trained to understand the imaging phenotypes, transcribe those observations (phenotypes) to correlate with underlying diseases and to characterize the images. However, in order to understand and characterize the molecular phenotype (to obtain genomic information) of solid heterogeneous tumours, the advanced sequencing of those tissues using biopsy is required. Thus, radiologists image the tissues from various views and angles in order to have the complete image phenotypes, thereby acquiring a huge amount of data. Deriving meaningful details from all these radiological data becomes challenging and raises the big data issues. Therefore, interest in the application of radiomics has been growing in recent years as it has the potential to provide significant interpretive and predictive information for decision support. Radiomics is a combination of conventional computer-aided diagnosis, deep learning methods, and human skills, and thus can be used for quantitative characterization of tumour phenotypes. This paper discusses the overview of radiomics workflow, the results of various radiomics-based studies conducted using various radiological images such as computed tomography (CT), magnetic resonance imaging (MRI), and positron-emission tomography (PET), the challenges we are facing, and the potential contribution of radiomics towards precision medicine.

Deep learning in bioinformatics

In the era of big data, transformation of biomedical big data into valuable knowledge has been one of the most important challenges in bioinformatics. Deep learning has advanced rapidly since the early 2000s and now demonstrates state-of-the-art performance in various fields. Accordingly, application of deep learning in bioinformatics to gain insight from data has been emphasized in both academia and industry. Here, we review deep learning in bioinformatics, presenting examples of current research. To provide a useful and comprehensive perspective, we categorize research both by the bioinformatics domain (i.e. omics, biomedical imaging, biomedical signal processing) and deep learning architecture (i.e. deep neural networks, convolutional neural networks, recurrent neural networks, emergent architectures) and present brief descriptions of each study. Additionally, we discuss theoretical and practical issues of deep learning in bioinformatics and suggest future research directions. We believe that this review will provide valuable insights and serve as a starting point for researchers to apply deep learning approaches in their bioinformatics studies.