Unfounded conclusions of equivalence in diagnostic accuracy studies: a pervasive fallacy of inference in pediatric radiology scientific abstracts

Quantification of hematoma and perihematomal edema volumes in intracerebral hemorrhage study: Design considerations in an artificial intelligence validation (QUANTUM) study

BACKGROUND

Hematoma and perihematomal edema volumes are important radiographic markers in spontaneous intracerebral hemorrhage. Accurate, reliable, and efficient quantification of these volumes will be paramount to their utility as measures of treatment effect in future clinical studies. Both manual and semi-automated quantification methods of hematoma and perihematomal edema volumetry are time-consuming and susceptible to inter-rater variability. Efforts are now underway to develop a fully automated algorithm that can replace them. A (QUANTUM) study to establish inter-quantification method measurement equivalency, which deviates from the traditional use of measures of agreement and a comparison hypothesis testing paradigm to indirectly infer quantification method measurement equivalence, is described in this article. The Quantification of Hematoma and Perihematomal Edema Volumes in Intracerebral Hemorrhage study aims to determine whether a fully automated quantification method and a semi-automated quantification method for quantification of hematoma and perihematomal edema volumes are equivalent to the hematoma and perihematomal edema volumes of the manual quantification method.

METHODS/DESIGN

Hematoma and perihematomal edema volumes of supratentorial intracerebral hemorrhage on 252 computed tomography scans will be prospectively quantified in random order by six raters using the fully automated, semi-automated, and manual quantification methods. Primary outcome measures for hematoma and perihematomal edema volumes will be quantified via computed tomography scan on admission (<24 h from symptom onset) and on day 3 (72 ± 12 h from symptom onset), respectively. Equivalence hypothesis testing will be conducted to determine if the hematoma and perihematomal edema volume measurements of the fully automated and semi-automated quantification methods are within 7.5% of the hematoma and perihematomal edema volume measurements of the manual quantification reference method.

DISCUSSION

By allowing direct equivalence hypothesis testing, the Quantification of Hematoma and Perihematomal Edema Volumes in Intracerebral Hemorrhage study offers advantages over radiology validation studies which utilize measures of agreement to indirectly infer measurement equivalence and studies which mistakenly try to infer measurement equivalence based on the failure of a comparison two-sided null hypothesis test to reach the significance level for rejection. The equivalence hypothesis testing paradigm applied to artificial intelligence application validation is relatively uncharted and warrants further investigation. The challenges encountered in the design of this study may influence future studies seeking to translate artificial intelligence medical technology into clinical practice.