Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework

Artificial intelligence (AI) software in radiology is becoming increasingly prevalent and performance is improving rapidly with new applications for given use cases being developed continuously, oftentimes with development and validation occurring in parallel. Several guidelines have provided reporting standards for publications of AI-based research in medicine and radiology. Yet, there is an unmet need for recommendations on the assessment of AI software before adoption and after commercialization. As the radiology AI ecosystem continues to grow and mature, a formalization of system assessment and evaluation is paramount to ensure patient safety, relevance and support to clinical workflows, and optimal allocation of limited AI development and validation resources before broader implementation into clinical practice. To fulfil these needs, we provide a glossary for AI software types, use cases and roles within the clinical workflow; list healthcare needs, key performance indicators and required information about software prior to assessment; and lay out examples of software performance metrics per software category. This conceptual framework is intended to streamline communication with the AI software industry and provide healthcare decision makers and radiologists with tools to assess the potential use of these software. The proposed software evaluation framework lays the foundation for a radiologist-led prospective validation network of radiology AI software. Learning Points: The rapid expansion of AI applications in radiology requires standardization of AI software specification, classification, and evaluation. The Canadian Association of Radiologists' AI Tech & Apps Working Group Proposes an AI Specification document format and supports the implementation of a clinical expert evaluation process for Radiology AI software.

Comparing Visual Scoring of Lung Injury with a Quantifying AI-Based Scoring in Patients with COVID-19

OBJECTIVES

Fast diagnosis of Coronavirus Disease 2019 (COVID-19), and the detection of high-risk patients are crucial but challenging in the pandemic outbreak. The aim of this study was to evaluate if deep learning-based software correlates well with the generally accepted visual-based scoring for quantification of the lung injury to help radiologist in triage and monitoring of COVID-19 patients.

MATERIALS AND METHODS

In this retrospective study, the lobar analysis of lung opacities (% opacities) by means of a prototype deep learning artificial intelligence (AI)-based software was compared to visual scoring. The visual scoring system used five categories (0: 0%, 1: 0-5%, 2: 5-25%, 3: 25-50%, 4: 50-75% and 5: >75% involvement). The total visual lung injury was obtained by the sum of the estimated grade of involvement of each lobe and divided by five.

RESULTS

The dataset consisted of 182 consecutive confirmed COVID-19 positive patients with a median age of 65 ± 16 years, including 110 (60%) men and 72 (40%) women. There was a correlation coefficient of 0.89 (p < 0.001) between the visual and the AI-based estimates of the severity of lung injury.

CONCLUSION

The study indicates a very good correlation between the visual scoring and AI-based estimates of lung injury in COVID-19.