MAIC-10 brief quality checklist for publications using artificial intelligence and medical images

Differences in technical and clinical perspectives on AI validation in cancer imaging: mind the gap!

Good practices in artificial intelligence (AI) model validation are key for achieving trustworthy AI. Within the cancer imaging domain, attracting the attention of clinical and technical AI enthusiasts, this work discusses current gaps in AI validation strategies, examining existing practices that are common or variable across technical groups (TGs) and clinical groups (CGs). The work is based on a set of structured questions encompassing several AI validation topics, addressed to professionals working in AI for medical imaging. A total of 49 responses were obtained and analysed to identify trends and patterns. While TGs valued transparency and traceability the most, CGs pointed out the importance of explainability. Among the topics where TGs may benefit from further exposure are stability and robustness checks, and mitigation of fairness issues. On the other hand, CGs seemed more reluctant towards synthetic data for validation and would benefit from exposure to cross-validation techniques, or segmentation metrics. Topics emerging from the open questions were utility, capability, adoption and trustworthiness. These findings on current trends in AI validation strategies may guide the creation of guidelines necessary for training the next generation of professionals working with AI in healthcare and contribute to bridging any technical-clinical gap in AI validation. RELEVANCE STATEMENT: This study recognised current gaps in understanding and applying AI validation strategies in cancer imaging and helped promote trust and adoption for interdisciplinary teams of technical and clinical researchers. KEY POINTS: Clinical and technical researchers emphasise interpretability, external validation with diverse data, and bias awareness in AI validation for cancer imaging. In cancer imaging AI research, clinical researchers prioritise explainability, while technical researchers focus on transparency and traceability, and see potential in synthetic datasets. Researchers advocate for greater homogenisation of AI validation practices in cancer imaging.

Evaluating the quality of radiomics-based studies for endometrial cancer using RQS and METRICS tools

OBJECTIVE

To assess the methodological quality of radiomics-based models in endometrial cancer using the radiomics quality score (RQS) and METhodological radiomICs score (METRICS).

METHODS

We systematically reviewed studies published by October 30th, 2023. Inclusion criteria were original radiomics studies on endometrial cancer using CT, MRI, PET, or ultrasound. Articles underwent a quality assessment by novice and expert radiologists using RQS and METRICS. The inter-rater reliability for RQS and METRICS among radiologists with varying expertise was determined. Subgroup analyses were performed to assess whether scores varied according to study topic, imaging technique, publication year, and journal quartile.

RESULTS

Sixty-eight studies were analysed, with a median RQS of 11 (IQR, 9-14) and METRICS score of 67.6% (IQR, 58.8-76.0); two different articles reached maximum RQS of 19 and METRICS of 90.7%, respectively. Most studies utilised MRI (82.3%) and machine learning methods (88.2%). Characterisation and recurrence risk stratification were the most explored outcomes, featured in 35.3% and 19.1% of articles, respectively. High inter-rater reliability was observed for both RQS (ICC: 0.897; 95% CI: 0.821, 0.946) and METRICS (ICC: 0.959; 95% CI: 0.928, 0.979). Methodological limitations such as lack of external validation suggest areas for improvement. At subgroup analyses, no statistically significant difference was noted.

CONCLUSIONS

Whilst using RQS, the quality of endometrial cancer radiomics research was apparently unsatisfactory, METRICS depicts a good overall quality. Our study highlights the need for strict compliance with quality metrics. Adhering to these quality measures can increase the consistency of radiomics towards clinical application in the pre-operative management of endometrial cancer.

CLINICAL RELEVANCE STATEMENT

Both the RQS and METRICS can function as instrumental tools for identifying different methodological deficiencies in endometrial cancer radiomics research. However, METRICS also reflected a focus on the practical applicability and clarity of documentation.

KEY POINTS

The topic of radiomics currently lacks standardisation, limiting clinical implementation. METRICS scores were generally higher than the RQS, reflecting differences in the development process and methodological content. A positive trend in METRICS score may suggest growing attention to methodological aspects in radiomics research.

Phantom evaluation of feasibility and applicability of artificial intelligence based pulmonary nodule detection in chest radiographs

The aim of our study was to evaluate the specific performance of an artificial intelligence (AI) algorithm for lung nodule detection in chest radiography for a larger number of nodules of different sizes and densities using a standardized phantom approach. A total of 450 nodules with varying density (d1 to d3) and size (3, 5, 8, 10 and 12 mm) were inserted in a Lungman phantom at various locations. Radiographic images with varying projections were acquired and processed using the AI algorithm for nodule detection. Computed tomography (CT) was performed for correlation. Ground truth (detectability) was established through a human consensus reading. Overall sensitivity and specificity of 0.978 and 0.812, respectively, were achieved for nodule detection. The false-positive rate was low with an overall rate of 0.19. The overall accuracy was calculated as 0.84 for all nodules. While most studies evaluating AI performance in the detection of pulmonary nodules have evaluated a mix of varying nodules, these are the first results of a controlled phantom-based study using a balanced number of nodules of all sizes and densities. To increase the radiologist's diagnostic performance and minimize the risk of decision bias, such algorithms have an obvious benefit in a clinical scenario.

Deep Convolutional Neural Network for Accurate Classification of Myofibroblastic Lesions on Patch-Based Images

OBJECTIVE

This study aimed to implement and evaluate a Deep Convolutional Neural Network for classifying myofibroblastic lesions into benign and malignant categories based on patch-based images.

METHODS

A Residual Neural Network (ResNet50) model, pre-trained with weights from ImageNet, was fine-tuned to classify a cohort of 20 patients (11 benign and 9 malignant cases). Following annotation of tumor regions, the whole-slide images (WSIs) were fragmented into smaller patches (224 × 224 pixels). These patches were non-randomly divided into training (308,843 patches), validation (43,268 patches), and test (42,061 patches) subsets, maintaining a 78:11:11 ratio. The CNN training was caried out for 75 epochs utilizing a batch size of 4, the Adam optimizer, and a learning rate of 0.00001.

RESULTS

ResNet50 achieved an accuracy of 98.97%, precision of 99.91%, sensitivity of 97.98%, specificity of 99.91%, F1 score of 98.94%, and AUC of 0.99.

CONCLUSIONS

The ResNet50 model developed exhibited high accuracy during training and robust generalization capabilities in unseen data, indicating nearly flawless performance in distinguishing between benign and malignant myofibroblastic tumors, despite the small sample size. The excellent performance of the AI model in separating such histologically similar classes could be attributed to its ability to identify hidden discriminative features, as well as to use a wide range of features and benefit from proper data preprocessing.

Ability of clinical data to predict readmission in Child and Adolescent Mental Health Services

This study addresses the challenge of predicting readmissions in Child and Adolescent Mental Health Services (CAMHS) by analyzing the predictability of readmissions over short, medium, and long term periods. Using health records spanning 35 years, which included 22,643 patients and 30,938 episodes of care, we focused on the episode of care as a central unit, defined as a referral-discharge cycle that incorporates assessments and interventions. Data pre-processing involved handling missing values, normalizing, and transforming data, while resolving issues related to overlapping episodes and correcting registration errors where possible. Readmission prediction was inferred from electronic health records (EHR), as this variable was not directly recorded. A binary classifier distinguished between readmitted and non-readmitted patients, followed by a multi-class classifier to categorize readmissions based on timeframes: short (within 6 months), medium (6 months - 2 years), and long (more than 2 years). Several predictive models were evaluated based on metrics like AUC, F1-score, precision, and recall, and the K-prototype algorithm was employed to explore similarities between episodes through clustering. The optimal binary classifier (Oversampled Gradient Boosting) achieved an AUC of 0.7005, while the multi-class classifier (Oversampled Random Forest) reached an AUC of 0.6368. The K-prototype resulted in three clusters as optimal (SI: 0.256, CI: 4473.64). Despite identifying relationships between care intensity, case complexity, and readmission risk, generalizing these findings proved difficult, partly because clinicians often avoid discharging patients likely to be readmitted. Overall, while this dataset offers insights into patient care and service patterns, predicting readmissions remains challenging, suggesting a need for improved analytical models that consider patient development, disease progression, and intervention effects.

Post-radiotherapy stage III/IV non-small cell lung cancer radiomics research: a systematic review and comparison of CLEAR and RQS frameworks

BACKGROUND

Lung cancer, the second most common cancer, presents persistently dismal prognoses. Radiomics, a promising field, aims to provide novel imaging biomarkers to improve outcomes. However, clinical translation faces reproducibility challenges, despite efforts to address them with quality scoring tools.

OBJECTIVE

This study had two objectives: 1) identify radiomics biomarkers in post-radiotherapy stage III/IV nonsmall cell lung cancer (NSCLC) patients, 2) evaluate research quality using the CLEAR (CheckList_for_EvaluAtion_of_Radiomics_research), RQS (Radiomics_Quality_Score) frameworks, and formulate an amalgamated CLEAR-RQS tool to enhance scientific rigor.

MATERIALS AND METHODS

A systematic literature review (Jun-Aug 2023, MEDLINE/PubMed/SCOPUS) was conducted concerning stage III/IV NSCLC, radiotherapy, and radiomic features (RF). Extracted data included study design particulars, such as sample size, radiotherapy/CT technique, selected RFs, and endpoints. CLEAR and RQS were merged into a CLEAR-RQS checklist. Three readers appraised articles utilizing CLEAR, RQS, and CLEAR-RQS metrics.

RESULTS

Out of 871 articles, 11 met the inclusion/exclusion criteria. The Median cohort size was 91 (range: 10-337) with 9 studies being single-center. No common RF were identified. The merged CLEAR-RQS checklist comprised 61 items. Most unreported items were within CLEAR's "methods" and "open-source," and within RQS's "phantom-calibration," "registry-enrolled prospective-trial-design," and "cost-effective-analysis" sections. No study scored above 50% on RQS. Median CLEAR scores were 55.74% (32.33/58 points), and for RQS, 17.59% (6.3/36 points). CLEAR-RQS article ranking fell between CLEAR and RQS and aligned with CLEAR.

CONCLUSION

Radiomics research in post-radiotherapy stage III/IV NSCLC exhibits variability and frequently low-quality reporting. The formulated CLEAR-RQS checklist may facilitate education and holds promise for enhancing radiomics research quality.

CLINICAL RELEVANCE STATEMENT

Current radiomics research in the field of stage III/IV postradiotherapy NSCLC is heterogenous, lacking reproducibility, with no identified imaging biomarker. Radiomics research quality assessment tools may enhance scientific rigor and thereby facilitate radiomics translation into clinical practice.

KEY POINTS

There is heterogenous and low radiomics research quality in postradiotherapy stage III/IV nonsmall cell lung cancer. Barriers to reproducibility are small cohort size, nonvalidated studies, missing technical parameters, and lack of data, code, and model sharing. CLEAR (CheckList_for_EvaluAtion_of_Radiomics_research), RQS (Radiomics_Quality_Score), and the amalgamated CLEAR-RQS tool are useful frameworks for assessing radiomics research quality and may provide a valuable resource for educational purposes in the field of radiomics.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Stroke Lesion Segmentation and Deep Learning: A Comprehensive Review

Stroke is a medical condition that affects around 15 million people annually. Patients and their families can face severe financial and emotional challenges as it can cause motor, speech, cognitive, and emotional impairments. Stroke lesion segmentation identifies the stroke lesion visually while providing useful anatomical information. Though different computer-aided software are available for manual segmentation, state-of-the-art deep learning makes the job much easier. This review paper explores the different deep-learning-based lesion segmentation models and the impact of different pre-processing techniques on their performance. It aims to provide a comprehensive overview of the state-of-the-art models and aims to guide future research and contribute to the development of more robust and effective stroke lesion segmentation models.

METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII

PURPOSE

To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.

METHODS

We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.

RESULT

In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.

CONCLUSION

In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.

CRITICAL RELEVANCE STATEMENT

A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.

KEY POINTS

• A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).

Towards reproducible radiomics research: introduction of a database for radiomics studies

OBJECTIVES

To investigate the model-, code-, and data-sharing practices in the current radiomics research landscape and to introduce a radiomics research database.

METHODS

A total of 1254 articles published between January 1, 2021, and December 31, 2022, in leading radiology journals (European Radiology, European Journal of Radiology, Radiology, Radiology: Artificial Intelligence, Radiology: Cardiothoracic Imaging, Radiology: Imaging Cancer) were retrospectively screened, and 257 original research articles were included in this study. The categorical variables were compared using Fisher's exact tests or chi-square test and numerical variables using Student's t test with relation to the year of publication.

RESULTS

Half of the articles (128 of 257) shared the model by either including the final model formula or reporting the coefficients of selected radiomics features. A total of 73 (28%) models were validated on an external independent dataset. Only 16 (6%) articles shared the data or used publicly available open datasets. Similarly, only 20 (7%) of the articles shared the code. A total of 7 (3%) articles both shared code and data. All collected data in this study is presented in a radiomics research database (RadBase) and could be accessed at https://github.com/EuSoMII/RadBase .

CONCLUSION

According to the results of this study, the majority of published radiomics models were not technically reproducible since they shared neither model nor code and data. There is still room for improvement in carrying out reproducible and open research in the field of radiomics.

CLINICAL RELEVANCE STATEMENT

To date, the reproducibility of radiomics research and open science practices within the radiomics research community are still very low. Ensuring reproducible radiomics research with model-, code-, and data-sharing practices will facilitate faster clinical translation.

KEY POINTS

• There is a discrepancy between the number of published radiomics papers and the clinical implementation of these published radiomics models. • The main obstacle to clinical implementation is the lack of model-, code-, and data-sharing practices. • In order to translate radiomics research into clinical practice, the radiomics research community should adopt open science practices.

Development of A Radiomic Model for MGMT Promoter Methylation Detection in Glioblastoma Using Conventional MRI

The methylation of the O6-methylguanine-DNA methyltransferase (MGMT) promoter is a molecular marker associated with a better response to chemotherapy in patients with glioblastoma (GB). Standard pre-operative magnetic resonance imaging (MRI) analysis is not adequate to detect MGMT promoter methylation. This study aims to evaluate whether the radiomic features extracted from multiple tumor subregions using multiparametric MRI can predict MGMT promoter methylation status in GB patients. This retrospective single-institution study included a cohort of 277 GB patients whose 3D post-contrast T1-weighted images and 3D fluid-attenuated inversion recovery (FLAIR) images were acquired using two MRI scanners. Three separate regions of interest (ROIs) showing tumor enhancement, necrosis, and FLAIR hyperintensities were manually segmented for each patient. Two machine learning algorithms (support vector machine (SVM) and random forest) were built for MGMT promoter methylation prediction from a training cohort (196 patients) and tested on a separate validation cohort (81 patients), based on a set of automatically selected radiomic features, with and without demographic variables (i.e., patients' age and sex). In the training set, SVM based on the selected radiomic features of the three separate ROIs achieved the best performances, with an average of 83.0% (standard deviation: 5.7%) for accuracy and 0.894 (0.056) for the area under the curve (AUC) computed through cross-validation. In the test set, all classification performances dropped: the best was obtained by SVM based on the selected features extracted from the whole tumor lesion constructed by merging the three ROIs, with 64.2% (95% confidence interval: 52.8-74.6%) accuracy and 0.572 (0.439-0.705) for AUC. The performances did not change when the patients' age and sex were included with the radiomic features into the models. Our study confirms the presence of a subtle association between imaging characteristics and MGMT promoter methylation status. However, further verification of the strength of this association is needed, as the low diagnostic performance obtained in this validation cohort is not sufficiently robust to allow clinically meaningful predictions.

The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study

BACKGROUND

Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables.

METHODS

This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis.

RESULTS

We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals.

CONCLUSIONS

The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.

Responsible AI practice and AI education are central to AI implementation: a rapid review for all medical imaging professionals in Europe

Artificial intelligence (AI) has transitioned from the lab to the bedside, and it is increasingly being used in healthcare. Radiology and Radiography are on the frontline of AI implementation, because of the use of big data for medical imaging and diagnosis for different patient groups. Safe and effective AI implementation requires that responsible and ethical practices are upheld by all key stakeholders, that there is harmonious collaboration between different professional groups, and customised educational provisions for all involved. This paper outlines key principles of ethical and responsible AI, highlights recent educational initiatives for clinical practitioners and discusses the synergies between all medical imaging professionals as they prepare for the digital future in Europe. Responsible and ethical AI is vital to enhance a culture of safety and trust for healthcare professionals and patients alike. Educational and training provisions for medical imaging professionals on AI is central to the understanding of basic AI principles and applications and there are many offerings currently in Europe. Education can facilitate the transparency of AI tools, but more formalised, university-led training is needed to ensure the academic scrutiny, appropriate pedagogy, multidisciplinarity and customisation to the learners' unique needs are being adhered to. As radiographers and radiologists work together and with other professionals to understand and harness the benefits of AI in medical imaging, it becomes clear that they are faced with the same challenges and that they have the same needs. The digital future belongs to multidisciplinary teams that work seamlessly together, learn together, manage risk collectively and collaborate for the benefit of the patients they serve.