Dynamic memory to alleviate catastrophic forgetting in continual learning with medical imaging

The Evolving Role of Artificial Intelligence in Radiotherapy Treatment Planning-A Literature Review

This paper examines the integration of artificial intelligence (AI) in radiotherapy for cancer treatment. The importance of radiotherapy in cancer management and its time-intensive planning process make AI adoption appealing especially with the escalating demand for radiotherapy. This review highlights the efficacy of AI across medical domains, where it surpasses human capabilities in areas such as cardiology and dermatology. Focusing on radiotherapy, the paper details AI's applications in target segmentation, dose optimization, and outcome prediction. It discusses adaptive radiotherapy's benefits and AI's potential to enhance patient outcomes with much improved treatment accuracy. The paper explores ethical concerns, including data privacy and bias, stressing the need for robust guidelines. Educating healthcare professionals and patients about AI's role is crucial as it acknowledges potential job-role changes and concerns about patients' trust in the use of AI. Overall, the integration of AI in radiotherapy holds transformative potential in streamlining processes, improving outcomes, and reducing costs. AI's potential to reduce healthcare costs underscores its significance with impactful change globally. However, successful implementation hinges on addressing ethical and logistical challenges and fostering collaboration among healthcare professionals and patient population data sets for its optimal utilization. Rigorous education, collaborative efforts, and global data sharing will be the compass guiding its' success in radiotherapy and healthcare.

Continual learning for seizure prediction via memory projection strategy

Despite extensive algorithms for epilepsy prediction via machine learning, most models are tailored for offline scenarios and cannot handle actual scenarios where data changes over time. Catastrophic forgetting(CF) for learned electroencephalogram(EEG) data occurs when EEG changes dynamically in the clinical setting. This paper implements a continual learning(CL) strategy Memory Projection(MP) for epilepsy prediction, which can be combined with other algorithms to avoid CF. Such a strategy enables the model to learn EEG data from each patient in dynamic subspaces with weak correlation layer by layer to minimize interference and promote knowledge transfer. Regularization Loss Reconstruction Algorithm and Matrix Dimensionality Reduction Algorithm are introduced into the core of MP. Experimental results show that MP exhibits excellent performance and low forgetting rates in sequential learning of seizure prediction. The forgetting rate of accuracy and sensitivity under multiple experiments are below 5%. When learning from multi-center datasets, the forgetting rates for accuracy and sensitivity decrease to 0.65% and 1.86%, making it comparable to state-of-the-art CL strategies. Through ablation experiments, we have analyzed that MP can operate with minimal storage and computational cost, which demonstrates practical potential for seizure prediction in clinical scenarios.

Deep learning with uncertainty estimation for automatic tumor segmentation in PET/CT of head and neck cancers: impact of model complexity, image processing and augmentation

Objective.Target volumes for radiotherapy are usually contoured manually, which can be time-consuming and prone to inter- and intra-observer variability. Automatic contouring by convolutional neural networks (CNN) can be fast and consistent but may produce unrealistic contours or miss relevant structures. We evaluate approaches for increasing the quality and assessing the uncertainty of CNN-generated contours of head and neck cancers with PET/CT as input.Approach.Two patient cohorts with head and neck squamous cell carcinoma and baseline18F-fluorodeoxyglucose positron emission tomography and computed tomography images (FDG-PET/CT) were collected retrospectively from two centers. The union of manual contours of the gross primary tumor and involved nodes was used to train CNN models for generating automatic contours. The impact of image preprocessing, image augmentation, transfer learning and CNN complexity, architecture, and dimension (2D or 3D) on model performance and generalizability across centers was evaluated. A Monte Carlo dropout technique was used to quantify and visualize the uncertainty of the automatic contours.Main results. CNN models provided contours with good overlap with the manually contoured ground truth (median Dice Similarity Coefficient: 0.75-0.77), consistent with reported inter-observer variations and previous auto-contouring studies. Image augmentation and model dimension, rather than model complexity, architecture, or advanced image preprocessing, had the largest impact on model performance and cross-center generalizability. Transfer learning on a limited number of patients from a separate center increased model generalizability without decreasing model performance on the original training cohort. High model uncertainty was associated with false positive and false negative voxels as well as low Dice coefficients.Significance.High quality automatic contours can be obtained using deep learning architectures that are not overly complex. Uncertainty estimation of the predicted contours shows potential for highlighting regions of the contour requiring manual revision or flagging segmentations requiring manual inspection and intervention.

Boosting knowledge diversity, accuracy, and stability via tri-enhanced distillation for domain continual medical image segmentation

Domain continual medical image segmentation plays a crucial role in clinical settings. This approach enables segmentation models to continually learn from a sequential data stream across multiple domains. However, it faces the challenge of catastrophic forgetting. Existing methods based on knowledge distillation show potential to address this challenge via a three-stage process: distillation, transfer, and fusion. Yet, each stage presents its unique issues that, collectively, amplify the problem of catastrophic forgetting. To address these issues at each stage, we propose a tri-enhanced distillation framework. (1) Stochastic Knowledge Augmentation reduces redundancy in knowledge, thereby increasing both the diversity and volume of knowledge derived from the old network. (2) Adaptive Knowledge Transfer selectively captures critical information from the old knowledge, facilitating a more accurate knowledge transfer. (3) Global Uncertainty-Guided Fusion introduces a global uncertainty view of the dataset to fuse the old and new knowledge with reduced bias, promoting a more stable knowledge fusion. Our experimental results not only validate the feasibility of our approach, but also demonstrate its superior performance compared to state-of-the-art methods. We suggest that our innovative tri-enhanced distillation framework may establish a robust benchmark for domain continual medical image segmentation.

Artificial intelligence in medical imaging is a tool for clinical routine and scientific discovery

The emergence of powerful machine learning methodology together with an increasing amount of data collected during clinical routine have fostered a growing role of artificial intelligence (AI) in medicine. Algorithms have become part of clinical care enhancing image reconstruction, detecting cancer or predicting individual risk to support treatment decisions and patient management. The entry into clinical care is determined by technological feasibility, integration into effective workflows, and immediacy of benefits. At the same time, research is advancing the integration of imaging data and other modalities such as genomics, and the linking of observations made at large scale with the understanding of underlying biological processes. AI will have impact in imaging and precision medicine not only because of the successful application of techniques established in other domains, but primarily because of the effective joint development of new technology and corresponding advance of diagnosis and care.

Closing the loop for AI-ready radiology

BACKGROUND

 In recent years, AI has made significant advancements in medical diagnosis and prognosis. However, the incorporation of AI into clinical practice is still challenging and under-appreciated. We aim to demonstrate a possible vertical integration approach to close the loop for AI-ready radiology.

METHOD

 This study highlights the importance of two-way communication for AI-assisted radiology. As a key part of the methodology, it demonstrates the integration of AI systems into clinical practice with structured reports and AI visualization, giving more insight into the AI system. By integrating cooperative lifelong learning into the AI system, we ensure the long-term effectiveness of the AI system, while keeping the radiologist in the loop.  RESULTS:  We demonstrate the use of lifelong learning for AI systems by incorporating AI visualization and structured reports. We evaluate Memory Aware-Synapses and Rehearsal approach and find that both approaches work in practice. Furthermore, we see the advantage of lifelong learning algorithms that do not require the storing or maintaining of samples from previous datasets.

CONCLUSION

 In conclusion, incorporating AI into the clinical routine of radiology requires a two-way communication approach and seamless integration of the AI system, which we achieve with structured reports and visualization of the insight gained by the model. Closing the loop for radiology leads to successful integration, enabling lifelong learning for the AI system, which is crucial for sustainable long-term performance.

KEY POINTS

  · The integration of AI systems into the clinical routine with structured reports and AI visualization.. · Two-way communication between AI and radiologists is necessary to enable AI that keeps the radiologist in the loop.. · Closing the loop enables lifelong learning, which is crucial for long-term, high-performing AI in radiology..

Relay learning: a physically secure framework for clinical multi-site deep learning

Big data serves as the cornerstone for constructing real-world deep learning systems across various domains. In medicine and healthcare, a single clinical site lacks sufficient data, thus necessitating the involvement of multiple sites. Unfortunately, concerns regarding data security and privacy hinder the sharing and reuse of data across sites. Existing approaches to multi-site clinical learning heavily depend on the security of the network firewall and system implementation. To address this issue, we propose Relay Learning, a secure deep-learning framework that physically isolates clinical data from external intruders while still leveraging the benefits of multi-site big data. We demonstrate the efficacy of Relay Learning in three medical tasks of different diseases and anatomical structures, including structure segmentation of retina fundus, mediastinum tumors diagnosis, and brain midline localization. We evaluate Relay Learning by comparing its performance to alternative solutions through multi-site validation and external validation. Incorporating a total of 41,038 medical images from 21 medical hosts, including 7 external hosts, with non-uniform distributions, we observe significant performance improvements with Relay Learning across all three tasks. Specifically, it achieves an average performance increase of 44.4%, 24.2%, and 36.7% for retinal fundus segmentation, mediastinum tumor diagnosis, and brain midline localization, respectively. Remarkably, Relay Learning even outperforms central learning on external test sets. In the meanwhile, Relay Learning keeps data sovereignty locally without cross-site network connections. We anticipate that Relay Learning will revolutionize clinical multi-site collaboration and reshape the landscape of healthcare in the future.

Cross Domain Lifelong Learning Based on Task Similarity

Humans gradually learn a sequence of cross-domain tasks and seldom experience catastrophic forgetting. In contrast, deep neural networks achieve good performance only in specific tasks within a single domain. To equip the network with lifelong learning capabilities, we propose a Cross-Domain Lifelong Learning (CDLL) framework that fully explores task similarities. Specifically, we employ a Dual Siamese Network (DSN) to learn the essential similarity features of tasks across different domains. To further understand similarity information across domains, we introduce a Domain-Invariant Feature Enhancement Module (DFEM) to better extract domain-invariant features. Moreover, we propose a Spatial Attention Network (SAN) that assigns different weights to various tasks based on the learned similarity features. Ultimately, to maximize the use of model parameters for learning new tasks, we propose a Structural Sparsity Loss (SSL) that can make the SAN as sparse as possible while ensuring accuracy. Experimental results show that our method effectively reduces catastrophic forgetting compared with state-of-the-art methods when continuously learning multiple tasks across different domains. It is worth noting that the proposed method scarcely forgets old knowledge while consistently enhancing the performance of learned tasks, more closely aligning with human learning.

Class incremental learning of remote sensing images based on class similarity distillation

When a well-trained model learns a new class, the data distribution differences between the new and old classes inevitably cause catastrophic forgetting in order to perform better in the new class. This behavior differs from human learning. In this article, we propose a class incremental object detection method for remote sensing images to address the problem of catastrophic forgetting caused by distribution differences among different classes. First, we introduce a class similarity distillation (CSD) loss based on the similarity between new and old class prototypes, ensuring the model's plasticity to learn new classes and stability to detect old classes. Second, to better extract class similarity features, we propose a global similarity distillation (GSD) loss that maximizes the mutual information between the new class feature and old class features. Additionally, we present a region proposal network (RPN)-based method that assigns positive and negative labels to prevent mislearning issues. Experiments demonstrate that our method is more accurate for class incremental learning on public DOTA and DIOR datasets and significantly improves training efficiency compared to state-of-the-art class incremental object detection methods.

Lifelong nnU-Net: a framework for standardized medical continual learning

As the enthusiasm surrounding Deep Learning grows, both medical practitioners and regulatory bodies are exploring ways to safely introduce image segmentation in clinical practice. One frontier to overcome when translating promising research into the clinical open world is the shift from static to continual learning. Continual learning, the practice of training models throughout their lifecycle, is seeing growing interest but is still in its infancy in healthcare. We present Lifelong nnU-Net, a standardized framework that places continual segmentation at the hands of researchers and clinicians. Built on top of the nnU-Net-widely regarded as the best-performing segmenter for multiple medical applications-and equipped with all necessary modules for training and testing models sequentially, we ensure broad applicability and lower the barrier to evaluating new methods in a continual fashion. Our benchmark results across three medical segmentation use cases and five continual learning methods give a comprehensive outlook on the current state of the field and signify a first reproducible benchmark.

Towards precision medicine based on a continuous deep learning optimization and ensemble approach

We developed a continuous learning system (CLS) based on deep learning and optimization and ensemble approach, and conducted a retrospective data simulated prospective study using ultrasound images of breast masses for precise diagnoses. We extracted 629 breast masses and 2235 images from 561 cases in the institution to train the model in six stages to diagnose benign and malignant tumors, pathological types, and diseases. We randomly selected 180 out of 3098 cases from two external institutions. The CLS was tested with seven independent datasets and compared with 21 physicians, and the system's diagnostic ability exceeded 20 physicians by training stage six. The optimal integrated method we developed is expected accurately diagnose breast masses. This method can also be extended to the intelligent diagnosis of masses in other organs. Overall, our findings have potential value in further promoting the application of AI diagnosis in precision medicine.

Automated detection and delineation of lymph nodes in haematoxylin & eosin stained digitised slides

Treatment of patients with oesophageal and gastric cancer (OeGC) is guided by disease stage, patient performance status and preferences. Lymph node (LN) status is one of the strongest prognostic factors for OeGC patients. However, survival varies between patients with the same disease stage and LN status. We recently showed that LN size from patients with OeGC might also have prognostic value, thus making delineations of LNs essential for size estimation and the extraction of other imaging biomarkers. We hypothesized that a machine learning workflow is able to: (1) find digital H&E stained slides containing LNs, (2) create a scoring system providing degrees of certainty for the results, and (3) delineate LNs in those images. To train and validate the pipeline, we used 1695 H&E slides from the OE02 trial. The dataset was divided into training (80%) and validation (20%). The model was tested on an external dataset of 826 H&E slides from the OE05 trial. U-Net architecture was used to generate prediction maps from which predefined features were extracted. These features were subsequently used to train an XGBoost model to determine if a region truly contained a LN. With our innovative method, the balanced accuracies of the LN detection were 0.93 on the validation dataset (0.83 on the test dataset) compared to 0.81 (0.81) on the validation (test) datasets when using the standard method of thresholding U-Net predictions to arrive at a binary mask. Our method allowed for the creation of an "uncertain" category, and partly limited false-positive predictions on the external dataset. The mean Dice score was 0.73 (0.60) per-image and 0.66 (0.48) per-LN for the validation (test) datasets. Our pipeline detects images with LNs more accurately than conventional methods, and high-throughput delineation of LNs can facilitate future LN content analyses of large datasets.

Mapping brain endophenotypes associated with idiopathic pulmonary fibrosis genetic risk

BACKGROUND

Idiopathic pulmonary fibrosis (IPF) is a serious disease of the lung parenchyma. It has a known polygenetic risk, with at least seventeen regions of the genome implicated to date. Growing evidence suggests linked multimorbidity of IPF with neurodegenerative or affective disorders. However, no study so far has explicitly explored links between IPF, associated genetic risk profiles, and specific brain features.

METHODS

We exploited imaging and genetic data from more than 32,000 participants available through the UK Biobank population-level resource to explore links between IPF genetic risk and imaging-derived brain endophenotypes. We performed a brain-wide imaging-genetics association study between the presence of 17 known IPF risk variants and 1248 multi-modal imaging-derived features, which characterise brain structure and function.

FINDINGS

We identified strong associations between cortical morphological features, white matter microstructure and IPF risk loci in chromosomes 17 (17q21.31) and 8 (DEPTOR). Through co-localisation analysis, we confirmed that cortical thickness in the anterior cingulate and more widespread white matter microstructure changes share a single causal variant with IPF at the chromosome 8 locus. Post-hoc preliminary analysis suggested that forced vital capacity may partially mediate the association between the DEPTOR variant and white matter microstructure, but not between the DEPTOR risk variant and cortical thickness.

INTERPRETATION

Our results reveal the associations between IPF genetic risk and differences in brain structure, for both cortex and white matter. Differences in tissue-specific imaging signatures suggest distinct underlying mechanisms with focal cortical thinning in regions with known high DEPTOR expression, unrelated to lung function, and more widespread microstructural white matter changes consistent with hypoxia or neuroinflammation with potential mediation by lung function.

FUNDING

This study was supported by the NIHR Nottingham Biomedical Research Centre and the UK Medical Research Council.

Assessing radiomics feature stability with simulated CT acquisitions

Medical imaging quantitative features had once disputable usefulness in clinical studies. Nowadays, advancements in analysis techniques, for instance through machine learning, have enabled quantitative features to be progressively useful in diagnosis and research. Tissue characterisation is improved via the “radiomics” features, whose extraction can be automated. Despite the advances, stability of quantitative features remains an important open problem. As features can be highly sensitive to variations of acquisition details, it is not trivial to quantify stability and efficiently select stable features. In this work, we develop and validate a Computed Tomography (CT) simulator environment based on the publicly available ASTRA toolbox (www.astra-toolbox.com). We show that the variability, stability and discriminative power of the radiomics features extracted from the virtual phantom images generated by the simulator are similar to those observed in a tandem phantom study. Additionally, we show that the variability is matched between a multi-center phantom study and simulated results. Consequently, we demonstrate that the simulator can be utilised to assess radiomics features’ stability and discriminative power.

Detection of metallic objects on digital radiographs with convolutional neural networks: A MRI screening tool

INTRODUCTION

Screening for metallic implants and foreign bodies before magnetic resonance imaging (MRI) examinations, are crucial for patient safety. History of health are supplied by the patient, a family member, screening of electronic health records or the picture and archive systems (PACS). PACS securely store and transmits digital radiographs (DR) and related reports with patient information. Convolutional neural networks (CNN) can be used to detect metallic objects in DRs stored in PACS. This study evaluates the accuracy of CNNs in the detection of metallic objects on DRs as an MRI screening tool.

METHODS

The musculoskeletal radiographs (MURA) dataset consisting of 14.863 upper extremity studies were stratified into datasets with and without metal. For each anatomical region: Elbow, finger, hand, humerus, forearm, shoulder and wrist we trained and validated CNN algorithms to classify radiographs with and without metal. Algorithm performance was evaluated with area under the receiver-operating curve (AUC), sensitivity, specificity, predictive values and accuracies compared with a reference standard of manually labelling.

RESULTS

Sensitivities, specificities and area under the ROC-curves (AUC) for the six anatomic regions ranged from 85.33% (95% CI: 78.64%-90.57%) to 100.00% (95% CI: 98.16%-100.00%), 75.44% (95% CI: 62.24%-85.87%) to 93.57% (95% CI: 88.78%-96.75%) and 0.95 to 0.99, respectively.

CONCLUSION

CNN algorithms classify DRs with metallic objects for six different anatomic regions with near-perfect accuracy. The rapid and iterative capability of the algorithms allows for scalable expansion and as a substitute MRI screening tool for metallic objects.

IMPLICATIONS FOR PRACTICE

All CNNs would be able to assist in metal detection of digital radiographs prior to MRI, an substantially decrease screening time.