From explanation to intervention: Interactive knowledge extraction from Convolutional Neural Networks used in radiology

Deep Learning models such as Convolutional Neural Networks (CNNs) are very effective at extracting complex image features from medical X-rays. However, the limited interpretability of CNNs has hampered their deployment in medical settings as they failed to gain trust among clinicians. In this work, we propose an interactive framework to allow clinicians to ask what-if questions and intervene in the decisions of a CNN, with the aim of increasing trust in the system. The framework translates a layer of a trained CNN into a measurable and compact set of symbolic rules. Expert interactions with visualizations of the rules promote the use of clinically-relevant CNN kernels and attach meaning to the rules. The definition and relevance of the kernels are supported by radiomics analyses and permutation evaluations, respectively. CNN kernels that do not have a clinically-meaningful interpretation are removed without affecting model performance. By allowing clinicians to evaluate the impact of adding or removing kernels from the rule set, our approach produces an interpretable refinement of the data-driven CNN in alignment with medical best practice.

The Scottish Medical Imaging Archive: 57.3 Million Radiology Studies Linked to Their Medical Records

Keywords: MRI, Imaging Sequences, Ultrasound, Mammography, CT, Angiography, Conventional Radiography Published under a CC BY 4.0 license. See also the commentary by Whitman and Vining in this issue.

Machine learning models in trusted research environments - understanding operational risks

Introduction

Trusted research environments (TREs) provide secure access to very sensitive data for research. All TREs operate manual checks on outputs to ensure there is no residual disclosure risk. Machine learning (ML) models require very large amount of data; if this data is personal, the TRE is a well-established data management solution. However, ML models present novel disclosure risks, in both type and scale.

Objectives

As part of a series on ML disclosure risk in TREs, this article is intended to introduce TRE managers to the conceptual problems and work being done to address them.

Methods

We demonstrate how ML models present a qualitatively different type of disclosure risk, compared to traditional statistical outputs. These arise from both the nature and the scale of ML modelling.

Results

We show that there are a large number of unresolved issues, although there is progress in many areas. We show where areas of uncertainty remain, as well as remedial responses available to TREs.

Conclusions

At this stage, disclosure checking of ML models is very much a specialist activity. However, TRE managers need a basic awareness of the potential risk in ML models to enable them to make sensible decisions on using TREs for ML model development.

Disclosure control of machine learning models from trusted research environments (TRE): New challenges and opportunities

Introduction

Artificial intelligence (AI) applications in healthcare and medicine have increased in recent years. To enable access to personal data, Trusted Research Environments (TREs) (otherwise known as Safe Havens) provide safe and secure environments in which researchers can access sensitive personal data and develop AI (in particular machine learning (ML)) models. However, currently few TREs support the training of ML models in part due to a gap in the practical decision-making guidance for TREs in handling model disclosure. Specifically, the training of ML models creates a need to disclose new types of outputs from TREs. Although TREs have clear policies for the disclosure of statistical outputs, the extent to which trained models can leak personal training data once released is not well understood.

Background

We review, for a general audience, different types of ML models and their applicability within healthcare. We explain the outputs from training a ML model and how trained ML models can be vulnerable to external attacks to discover personal data encoded within the model.

Risks

We present the challenges for disclosure control of trained ML models in the context of training and exporting models from TREs. We provide insights and analyse methods that could be introduced within TREs to mitigate the risk of privacy breaches when disclosing trained models.

Discussion

Although specific guidelines and policies exist for statistical disclosure controls in TREs, they do not satisfactorily address these new types of output requests; i.e., trained ML models. There is significant potential for new interdisciplinary research opportunities in developing and adapting policies and tools for safely disclosing ML outputs from TREs.

Next-Generation Capabilities in Trusted Research Environments: Interview Study

BACKGROUND

A Trusted Research Environment (TRE; also known as a Safe Haven) is an environment supported by trained staff and agreed processes (principles and standards), providing access to data for research while protecting patient confidentiality. Accessing sensitive data without compromising the privacy and security of the data is a complex process.

OBJECTIVE

This paper presents the security measures, administrative procedures, and technical approaches adopted by TREs.

METHODS

We contacted 73 TRE operators, 22 (30%) of whom, in the United Kingdom and internationally, agreed to be interviewed remotely under a nondisclosure agreement and to complete a questionnaire about their TRE.

RESULTS

We observed many similar processes and standards that TREs follow to adhere to the Seven Safes principles. The security processes and TRE capabilities for supporting observational studies using classical statistical methods were mature, and the requirements were well understood. However, we identified limitations in the security measures and capabilities of TREs to support "next-generation" requirements such as wide ranges of data types, ability to develop artificial intelligence algorithms and software within the environment, handling of big data, and timely import and export of data.

CONCLUSIONS

We found a lack of software or other automation tools to support the community and limited knowledge of how to meet the next-generation requirements from the research community. Disclosure control for exporting artificial intelligence algorithms and software was found to be particularly challenging, and there is a clear need for additional controls to support this capability within TREs.

Scottish Medical Imaging Service - Technical and Governance controls

ObjectivesThe Scottish Medical Imaging (SMI) service provides linkable, population based, “research-ready” real-world medical images for researchers to develop or validate AI algorithms within the Scottish National Safe Haven. The PICTURES research programme is developing novel methods to enhance the SMI service offering through research in cybersecurity and software/data/infrastructure engineering. ApproachAdditional technical and governance controls were required to enable safe access to medical images. The researcher is isolated from the rest of the trusted research environment (TRE) using a Project Private Zone (PPZ). This enables researchers to build and install their own software stack, and protects the TRE from malicious code. Guidelines are under development for researchers on the safe development of algorithms and the expected relationship between the size of the model and the training dataset. There is associated work on the statistical disclosure control of models to enable safe release of trained models from the TRE. ResultsA policy enabling the use of “Non-standard software” based on prior research, domain knowledge and experience gained from two contrasting research studies was developed.  Additional clauses have been added to the legal control – the eDRIS User Agreement – signed by each researcher and their Head of Department.  Penalties for attempting to import or use malware, remove data within models or any attempt to deceive or circumvent such controls are severe, and apply to both the individual and their institution. The process of building and deploying a PPZ has been developed allowing researchers to install their own software. No attempt has yet been made to add additional ethical controls; however, a future service development could be validating the performance of researchers’ algorithms on our training dataset. ConclusionThe availability to conduct research using images poses new challenges and risks for those commissioning and operating TREs. The Private Project Zone and our associated governance controls are a huge step towards supporting the needs of researchers in the 21st century.

Generalisation and robustness investigation for facial and speech emotion recognition using bio-inspired spiking neural networks

Emotion recognition through facial expression and non-verbal speech represents an important area in affective computing. They have been extensively studied from classical feature extraction techniques to more recent deep learning approaches. However, most of these approaches face two major challenges: (1) robustness—in the face of degradation such as noise, can a model still make correct predictions? and (2) cross-dataset generalisation—when a model is trained on one dataset, can it be used to make inference on another dataset?. To directly address these challenges, we first propose the application of a spiking neural network (SNN) in predicting emotional states based on facial expression and speech data, then investigate, and compare their accuracy when facing data degradation or unseen new input. We evaluate our approach on third-party, publicly available datasets and compare to the state-of-the-art techniques. Our approach demonstrates robustness to noise, where it achieves an accuracy of 56.2% for facial expression recognition (FER) compared to 22.64% and 14.10% for CNN and SVM, respectively, when input images are degraded with the noise intensity of 0.5, and the highest accuracy of 74.3% for speech emotion recognition (SER) compared to 21.95% of CNN and 14.75% for SVM when audio white noise is applied. For generalisation, our approach achieves consistently high accuracy of 89% for FER and 70% for SER in cross-dataset evaluation and suggests that it can learn more effective feature representations, which lead to good generalisation of facial features and vocal characteristics across subjects.