Zaazoue KA, McCann MR, Ahmed AK, Cortopassi IO, Erben YM, Little BP, Stowell JT, Toskich BB, Ritchie CA. Evaluating the Performance of a Commercially Available AI Algorithm for Automated Detection of Pulmonary Embolism on CECT and CTPA of COVID-19 Patients.
Mayo Clin Proc Innov Qual Outcomes 2023;
7:143-152. [PMID:
37020901 PMCID:
PMC9995315 DOI:
10.1016/j.mayocpiqo.2023.03.001]
[Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/11/2023] Open
Abstract
Objective
To investigate the performance of a commercially available artificial intelligence (AI) algorithm for detection of pulmonary embolism (PE) on contrast-enhanced CTs in patients hospitalized for COVID-19.
Patients & Methods
Retrospective analysis was performed of all contrast-enhanced chest CTs on patients admitted for COVID-19 between March 2020 and December 2021. Based on the original radiology reports, all PE-positive exams were included (n=527). Using a reversed flow single gate diagnostic accuracy case-control model, a randomly selected cohort of PE-negative exams (n=977) was included. Pulmonary parenchymal disease severity was assessed for all included studies using a semi-quantitative system, the Total Severity Score (TSS). All included CTs were sent for interpretation by the commercially available AI algorithm, Aidoc. Discrepancies between AI and original radiology reports were resolved by three blinded radiologists, who rendered a final determination of indeterminate, positive, or negative.
Results
A total of 78 studies were found to be discrepant, of which 13 (16.6%) were deemed indeterminate by readers and excluded. The sensitivity and specificity of AI was 93.2%; (95% confidence interval [CI] 90.6-95.2%), and 99.6%; (95% CI 98.9-99.9%), respectively. AI's accuracy for all TSS groups (mild, moderate, severe) was high (98.4%, 96.7%, and 97.2%, respectively). AI was more accurate in PE detection on CTPAs vs CECTs (P < .001), with optimal HU of 362 (P=.048).
Conclusion
The AI algorithm demonstrated high sensitivity, specificity, and accuracy for PE on contrast enhanced CTs in COVID-19 patients regardless of parenchymal disease. Accuracy was significantly affected by the mean attenuation of the pulmonary vasculature. How this affects the legitimacy of the binary outcomes reported by AI is not yet known.
Collapse