Evidence of a cognitive bias in the quantification of COVID-19 with CT: an artificial intelligence randomised clinical trial

Availability and transparency of artificial intelligence models in radiology: a meta-research study

OBJECTIVES

This meta-research study explored the availability of artificial intelligence (AI) models from development studies published in leading radiology journals in 2022, with availability defined as the transparent reporting of relevant technical details, such as model architecture and weights, necessary for independent replication.

MATERIALS AND METHODS

A systematic search of Ovid Medline and Embase was conducted to identify AI model development studies published in five leading radiology journals in 2022. Data were extracted on study characteristics, model details, and code and model-sharing practices. The proportion of AI studies sharing their models was analyzed. Logistic regression analyses were employed to explore associations between study characteristics and model availability.

RESULTS

Of 268 studies reviewed, 39.9% (n = 107) made their models available. Deep learning (DL) models exhibited particularly low availability, with only 11.5% (n = 13) of the 113 studies being fully available. Training codes for DL models were provided in 22.1% (n = 25), suggesting limited ability to train DL models with one's own data. Multivariable logistic regression analysis showed that the use of traditional regression-based models (odds ratio [OR], 17.11; 95% CI: 5.52, 53.05; p < 0.001) was associated with higher availability, while the radiomics package usage (OR, 0.27; 95% CI: 0.11, 0.65; p = 0.003) was associated with lower availability.

CONCLUSION

The availability of AI models in radiology publications remains suboptimal, especially for DL models. Enforcing model-sharing policies, enhancing external validation platforms, addressing commercial restrictions, and providing demos for commercial models in open repositories are necessary to improve transparency and replicability in radiology AI research.

KEY POINTS

Question The study addresses the limited availability of AI models in radiology, especially DL models, which impacts external validation and clinical reliability. Findings Only 39.9% of radiology AI studies made their models available, with DL models showing particularly low availability at 11.5%. Clinical relevance Improving the availability of radiology AI models is essential for enabling external validation, ensuring reliable clinical application, and advancing patient care by fostering robust and transparent AI systems.

Digitalised multidisciplinary conferences effectively identify and prevent imaging-related medical error in intensive care patients during the COVID-19 pandemic

This study aims to assess the effectiveness of digital multidisciplinary conferences (MDCs) in preventing imaging-related quality management (QM) events during the coronavirus-disease-19 (COVID-19) pandemic. COVID-19 challenged interdisciplinary exchange and QM measures for patient safety. Regular MDCs between radiologists and intensive care unit (ICU) physicians, introduced in our hospital in 2018, enable re-evaluation of imaging examinations and bilateral feedback. MDC protocols from 2020 to 2021 were analysed regarding imaging-related QM events. Epidemiological data on COVID-19 were matched with MDCs. 333 MDCs including 1324 radiological examinations in 857 patients (median age = 64 (IQR = 55-73) years, 66.7% male) were analysed. MDCs were held within a median of 1 day after imaging (IQR = 1-3). QM events were identified in 2.7% (n = 36/1324) of examinations. This represented a significant decrease compared to a control group from 2018/2019 (QM events identified in 14.0%, p < 0.001). QM incidence remained consistent in the pandemic cohort (regression coefficient estimate = -0.01, 95% confidence interval = [0.000, 0.000], p = 0.68). 81% (n = 29/36) of QM events were report-related, 19% process-related (n = 6/36), and 2.8% indication-related (n = 1/36). In 7.3% (n = 97/1324) of examinations, the patient was affected by COVID-19. With MDCs as an effective feedback mechanism in place, the challenges of the COVID-19 pandemic led to no increase in QM incidence. Notably, COVID status did not impact QM event occurrence.

Evaluation of Effectiveness of Self-Supervised Learning in Chest X-Ray Imaging to Reduce Annotated Images

A significant challenge in machine learning-based medical image analysis is the scarcity of medical images. Obtaining a large number of labeled medical images is difficult because annotating medical images is a time-consuming process that requires specialized knowledge. In addition, inappropriate annotation processes can increase model bias. Self-supervised learning (SSL) is a type of unsupervised learning method that extracts image representations. Thus, SSL can be an effective method to reduce the number of labeled images. In this study, we investigated the feasibility of reducing the number of labeled images in a limited set of unlabeled medical images. The unlabeled chest X-ray (CXR) images were pretrained using the SimCLR framework, and then the representations were fine-tuned as supervised learning for the target task. A total of 2000 task-specific CXR images were used to perform binary classification of coronavirus disease 2019 (COVID-19) and normal cases. The results demonstrate that the performance of pretraining on task-specific unlabeled CXR images can be maintained when the number of labeled CXR images is reduced by approximately 40%. In addition, the performance was significantly better than that obtained without pretraining. In contrast, a large number of pretrained unlabeled images are required to maintain performance regardless of task specificity among a small number of labeled CXR images. In summary, to reduce the number of labeled images using SimCLR, we must consider both the number of images and the task-specific characteristics of the target images.

Artificial Intelligence Applied to Chest X-ray: A Reliable Tool to Assess the Differential Diagnosis of Lung Pneumonia in the Emergency Department

BACKGROUND

Considering the large number of patients with pulmonary symptoms admitted to the emergency department daily, it is essential to diagnose them correctly. It is necessary to quickly solve the differential diagnosis between COVID-19 and typical bacterial pneumonia to address them with the best management possible. In this setting, an artificial intelligence (AI) system can help radiologists detect pneumonia more quickly.

METHODS

We aimed to test the diagnostic performance of an AI system in detecting COVID-19 pneumonia and typical bacterial pneumonia in patients who underwent a chest X-ray (CXR) and were admitted to the emergency department. The final dataset was composed of three sub-datasets: the first included all patients positive for COVID-19 pneumonia (n = 1140, namely "COVID-19+"), the second one included all patients with typical bacterial pneumonia (n = 500, "pneumonia+"), and the third one was composed of healthy subjects (n = 1000). Two radiologists were blinded to demographic, clinical, and laboratory data. The developed AI system was used to evaluate all CXRs randomly and was asked to classify them into three classes. Cohen's κ was used for interrater reliability analysis. The AI system's diagnostic accuracy was evaluated using a confusion matrix, and 95%CIs were reported as appropriate.

RESULTS

The interrater reliability analysis between the most experienced radiologist and the AI system reported an almost perfect agreement for COVID-19+ (κ = 0.822) and pneumonia+ (κ = 0.913). We found 96% sensitivity (95% CIs = 94.9-96.9) and 79.8% specificity (76.4-82.9) for the radiologist and 94.7% sensitivity (93.4-95.8) and 80.2% specificity (76.9-83.2) for the AI system in the detection of COVID-19+. Moreover, we found 97.9% sensitivity (98-99.3) and 88% specificity (83.5-91.7) for the radiologist and 97.5% sensitivity (96.5-98.3) and 83.9% specificity (79-87.9) for the AI system in the detection of pneumonia+ patients. Finally, the AI system reached an accuracy of 93.8%, with a misclassification rate of 6.2% and weighted-F1 of 93.8% in detecting COVID+, pneumonia+, and healthy subjects.

CONCLUSIONS

The AI system demonstrated excellent diagnostic performance in identifying COVID-19 and typical bacterial pneumonia in CXRs acquired in the emergency setting.