Accuracy and reproducibility of visual coronary stenosis estimates using information from multiple observers

CathAI: fully automated coronary angiography interpretation and stenosis estimation

Coronary angiography is the primary procedure for diagnosis and management decisions in coronary artery disease (CAD), but ad-hoc visual assessment of angiograms has high variability. Here we report a fully automated approach to interpret angiographic coronary artery stenosis from standard coronary angiograms. Using 13,843 angiographic studies from 11,972 adult patients at University of California, San Francisco (UCSF), between April 1, 2008 and December 31, 2019, we train neural networks to accomplish four sequential necessary tasks for automatic coronary artery stenosis localization and estimation. Algorithms are internally validated against criterion-standard labels for each task in hold-out test datasets. Algorithms are then externally validated in real-world angiograms from the University of Ottawa Heart Institute (UOHI) and also retrained using quantitative coronary angiography (QCA) data from the Montreal Heart Institute (MHI) core lab. The CathAI system achieves state-of-the-art performance across all tasks on unselected, real-world angiograms. Positive predictive value, sensitivity and F1 score are all ≥90% to identify projection angle and ≥93% for left/right coronary artery angiogram detection. To predict obstructive CAD stenosis (≥70%), CathAI exhibits an AUC of 0.862 (95% CI: 0.843-0.880). In UOHI external validation, CathAI achieves AUC 0.869 (95% CI: 0.830-0.907) to predict obstructive CAD. In the MHI QCA dataset, CathAI achieves an AUC of 0.775 (95%. CI: 0.594-0.955) after retraining. In conclusion, multiple purpose-built neural networks can function in sequence to accomplish automated analysis of real-world angiograms, which could increase standardization and reproducibility in angiographic coronary stenosis assessment.

Comparison of Physician Visual Assessment With Quantitative Coronary Angiography in Assessment of Stenosis Severity in China

IMPORTANCE

Although physician visual assessment (PVA) of stenosis severity is a standard clinical practice to support decisions for coronary revascularization, there are concerns about its accuracy.

OBJECTIVE

To compare PVA with quantitative coronary angiography (QCA) as a means of assessing stenosis severity among patients undergoing percutaneous coronary intervention (PCI) in China.

DESIGN, SETTING, AND PARTICIPANTS

A cross-sectional study (2012-2013) of a random subset of 1295 patients from the China Patient-centered Evaluative Assessment of Cardiac Events (PEACE) Prospective PCI Study was carried out. The PEACE Prospective PCI study recruited a consecutive sample of patients undergoing PCI at 35 hospitals in 18 provinces of China. The coronary angiograms of this subset of participants were reviewed using QCA by 2 independent core laboratories blinded to PVA readings.

MAIN OUTCOMES AND MEASURES

Differences between PVA and QCA assessments of stenosis severity for lesions for which PCI was performed and variation of these differences among hospitals and physicians, stratified by the diagnosis of acute myocardial infarction (AMI).

RESULTS

In patients without AMI, the mean (SD) age was 62 (10) years, and 217 (31.5%) were women; in patients with AMI, the mean (SD) age was 60 (11) years, and 153 (25.2%) were women. The mean (SD) percent diameter stenosis by PVA was 16.0% (11.5%) greater than that by QCA in patients without AMI and 10.2% (12.3%) in those with AMI (P < .001 for both comparisons). In patients without AMI, of 837 lesions with 70% or more stenosis by PVA, 427 (50.6%) were less than 70% by QCA; in patients with AMI, similar patterns were observed to a lesser extent. Among patients without AMI, only 4 (0.47%) lesions were additionally assessed with fractional flow reserve. Among 30 hospitals, the difference between PVA and QCA readings of stenosis severity varied from 7.6% (95% CI, 0.4%-14.7%) to 21.3% (95% CI, 17.1%-24.9%) among non-AMI patients. Across 57 physicians, this difference varied from 6.9% (95% CI, -1.4%-15.3%) to 26.4% (95% CI, 21.5%-31.4%).

CONCLUSIONS AND RELEVANCE

For coronary lesions treated with PCI in China, PVA reported substantially higher readings of stenosis severity than QCA, with large variation across hospitals and physicians. These findings highlight the need to improve the accuracy of information used to guide treatment decisions in catheterization laboratories.

Comparison of clinical interpretation with visual assessment and quantitative coronary angiography in patients undergoing percutaneous coronary intervention in contemporary practice: the Assessing Angiography (A2) project

BACKGROUND

Studies conducted decades ago described substantial disagreement and errors in physicians' angiographic interpretation of coronary stenosis severity. Despite the potential implications of such findings, no large-scale efforts to measure or improve clinical interpretation were subsequently undertaken.

METHODS AND RESULTS

We compared clinical interpretation of stenosis severity in coronary lesions with an independent assessment using quantitative coronary angiography (QCA) in 175 randomly selected patients undergoing elective percutaneous coronary intervention at 7 US hospitals in 2011. To assess agreement, we calculated mean difference in percent diameter stenosis between clinical interpretation and QCA and a Cohen weighted κ statistic. Of 216 treated lesions, median percent diameter stenosis was 80.0% (quartiles 1 and 3, 80.0% and 90.0%), with 213 (98.6%) assessed as ≥70%. Mean difference in percent diameter stenosis between clinical interpretation and QCA was 8.2±8.4%, reflecting an average higher percent diameter stenosis by clinical interpretation (P<0.001). A weighted κ of 0.27 (95% confidence interval, 0.18-0.36) was found between the 2 measurements. Of 213 lesions considered ≥70% by clinical interpretation, 56 (26.3%) were <70% by QCA, although none were <50%. Differences between the 2 measurements were largest for intermediate lesions by QCA (50% to <70%), with variation existing across sites.

CONCLUSIONS

Physicians tended to assess coronary lesions treated with percutaneous coronary intervention as more severe than measurements by QCA. Almost all treated lesions were ≥70% by clinical interpretation, whereas approximately one quarter were <70% by QCA. These findings suggest opportunities to improve clinical interpretation of coronary angiography.

Potential shift from coronary bypass surgery to percutaneous coronary intervention for multivessel disease and its economic impact in the drug-eluting stent era

BACKGROUND

Drug-eluting stents (DES) may promote percutaneous coronary intervention (PCI) procedures in patients traditionally referred for coronary artery bypass graft (CABG) surgery and may save money.

OBJECTIVES

The purpose of the present study was to quantify the potential shift from CABG surgery to multivessel PCI in the DES era and to model the economic consequences.

METHODS

Based on predefined criteria, the feasibility of PCI was evaluated in patients with multivessel coronary artery disease who underwent CABG surgery before the availability of DES at the Centre Hospitalier de l'Université de Montréal's Notre-Dame Hospital (Montreal, Quebec). Modelling was used to evaluate the potential cost savings using multivessel PCI instead of CABG surgery. Equal one-year outcomes in both groups were assumed, with the exception of a 10% repeat revascularization (RR) rate in the DES group and a 4% RR rate in the CABG group. The impact of those assumptions was evaluated using 1000 Monte Carlo simulations.

RESULTS

The authors retrospectively evaluated that, of 289 patients who underwent CABG without concomitant valve surgery between January and December 2003, only 22 patients (8%) were good candidates for multivessel DES implantation. The procedures would have involved an average of 3.6 DES per patient. The average cost per revascularization procedure was $14,402 with surgery and $11,220 for multivessel DES implantation (using $2,200 DES), leading to a savings of $3,182 per patient. However, after including RR procedures, PCI would only have been associated with savings of $812 per surgery avoided. Monte Carlo analysis revealed that surgery may be less expensive than PCI in 36% of patients.

CONCLUSIONS

Most patients who underwent CABG surgery in 2003 were retrospectively judged to be ineligible for multivessel PCI with DES. In the rare eligible patient, multivessel PCI with DES is not expected to produce savings to health care costs in Canada unless the DES purchase cost continues to decrease.

Angiographic Assessment of Initial Balloon Angioplasty Results

PURPOSE

To determine the influence of three factors involved in the angiographic assessment of balloon angioplasty-interobserver variability, operator bias, and the definition used to determine success-on the primary (technical) results of angioplasty in the peripheral arteries.

MATERIALS AND METHODS

Percent stenosis in 107 lesions in lower-extremity arteries was graded by three independent, experienced vascular radiologists ("observers") before and after balloon angioplasty and their estimates were compared with the initial interpretations reported by the physician performing the procedure ("operator") and an automated quantitative computer analysis. Observer variability was measured with use of intraclass correlation coefficients and SD. Differences among the operator, observers, and the computer were analyzed with use of the Wilcoxon signed-rank test and analysis of variance. For each evaluator, the results in this series of lesions were interpreted with three different definitions of success.

RESULTS

Estimation of residual stenosis varied by an average range of 22.76% with an average SD of 8.99. The intraclass correlation coefficients averaged 0.59 for residual stenosis after angioplasty for the three observers but decreased to 0.36 when the operator was included as the fourth evaluator. There was good to very good agreement among the three independent observers and the computer, but poor correlation with the operator (P </= .001). The primary success rates for this series of lesions varied from a low of 47% to high of 99%, depending solely on which definition of success was used. Significant differences among the operator, the three observers, and the computer were not present when the definition of success was based on less than 50% residual stenosis.

CONCLUSIONS

Observer variability and bias in the subjective evaluation of peripheral angioplasty can have a significant influence on the reported initial success rates. This effect can be largely eliminated with the use of residual stenosis of less than 50% to define success. Otherwise, meaningful evaluation of angioplasty results will require independent panels of evaluators or computerized measurements.

Estimation of myocardial hemodynamics before and after intervention in children with kawasaki disease

OBJECTIVES

We used myocardial fractional flow reserve (FFR(myo)) and coronary flow reserve (CFR) to estimate cut-off values for assessment of the functional severity of coronary stenosis and myocardial ischemia, and we tested the usefulness of coronary blood hemodynamic measurements before and after plain old balloon angioplasty (POBA) and coronary artery bypass graft surgery (CABG).

BACKGROUND

Fractional flow reserve and CFR are useful for assessing the functional severity of coronary artery stenosis, coronary microvascular dysfunction, and myocardial ischemia during cardiac catheterization in adults. However, there have been no reports on the use of these measurements in children with Kawasaki disease (KD).

METHODS

The study group included 128 patients with 314 coronary branches. The subjects were classified into three groups: normal coronary group, with 206 branches; abnormal coronary artery without ischemia group, with 58 branches; and ischemia group, with 50 branches.

RESULTS

In each branch, CFR and FFR(myo) were significantly lower in the ischemia group than in the other groups. Cut-off values for assessing the functional severity of coronary stenosis and CFR were approximately equal to those obtained for adults (CFR: <2.0; FFR(myo): <0.75). We obtained very high sensitivity and specificity for estimating myocardial ischemia using CFR and FFR(myo) (CFR: 94.0% and 98.5%, respectively; FFR(myo): 95.7% and 99.1%, respectively). Both CFR and FFR(myo) were reliable indicators of coronary hemodynamics before and after POBA and CABG.

CONCLUSIONS

Together, CFR and FFR(myo) provide a useful index for assessing the functional severity of coronary artery stenosis and myocardial ischemia and estimating the effectiveness of POBA and CABG in children with KD, the same as they do for adults.

Magnitude and consequences of error in coronary angiography interpretation (the ACRE study)

In the routine reporting of coronary angiograms, there are no contemporary estimates of the magnitude and consequences of interobserver variability. We therefore measured the agreement beyond chance between (1) the number of narrowed arteries on an angiographic report extracted from case notes and independent assessments by 2 cardiologists, and (2) actual patient management over an 18-month follow-up period and each cardiologist's hypothetical management proposal based on abstracted clinical details. Two hundred nine angiograms were randomly selected from 4,121 patients in a prospective study (Appropriateness of Coronary Revascularisation [ACRE study]). The number of narrowed arteries was defined using Coronary Artery Surgery Study (CASS) criteria. For the number of narrowed arteries, cardiologists A and B agreed with the angiographic report in 126 patients (60%, weighted kappa = 0.64) and 124 patients (59%, weighted kappa = 0.63), respectively. In a subset of 92 patients (44%) there was unanimous agreement on the number of narrowed arteries (both cardiologists agreed with the angiographic report). Comparing actual management (34 percutaneous transluminal coronary angioplasty and 39 coronary artery bypass grafting procedures on follow-up) with each of the cardiologist's management recommendations showed agreement in 150 patients (72%, kappa = 0.46) and 154 patients (74%, kappa = 0.48) for cardiologists A and B, respectively. These agreements on management improved (p = 0.05) for cardiologist B (but not A) when analysis was confined to the subset of 92 patients, showing agreement in 73 patients (79%, kappa = 0.60). Thus, in routine clinical practice, the agreement beyond chance in interpretation of the number of narrowed arteries was good. Disagreements on subsequent patient management arose as a result of, and independent of, errors in angiographic interpretation.

Interpretation of renal angiography by radiologists

OBJECTIVE

Different radiologists may show considerable variation in their interpretation of renal angiograms. We therefore wished to establish the reliability of their interpretation.

DESIGN

Assessment of the intra- and inter-observer agreement of the interpretation of renal angiograms.

SETTING

Tertiary referral university hospital.

PATIENTS

Hypertensive patients suspected of renovascular hypertension on clinical grounds or on the basis of renography.

INTERVENTIONS

Patients were prospectively selected to undergo a renal angiography via the femoral approach.

MAIN OUTCOME MEASURES

Intra- and inter-observer agreement of the degree and site of stenosis.

RESULTS

The difference between two estimates of the degree of stenosis ranged from 0 to 65% for the individual readers and from 0 to 75% between two readers. When the site of greatest stenosis was in the origin of the renal artery, the intra-observer agreement kappa ranged from 0.54-0.71, the inter-observer agreement across multiple readers being 0.43. In a post hoc analysis using two different cut-off points of stenosis (50 or 70%), the intra- and inter-observer agreement was better at the 70% cut-off-point. In a subset of patients with stenosis and a renin ratio greater than 1.5, both the intra- and inter-observer agreement were much better than when all angiograms were considered.

CONCLUSIONS

Assessment of the diagnostic performance of three experienced radiologists in their interpretation of renal artery angiograms indicates that the intra- and inter-observer agreement with respect to their estimates of the degree of stenosis and the site of greatest stenosis are rather poor but their diagnostic performance improves in patients with stenosis and a renin ratio greater than 1.5. There is a need for more objective assessment of renal artery lesions.

Appropriateness of referral of coronary angiography patients in Sweden. SECOR/SBU Project Group

OBJECTIVE

To evaluate the appropriateness of referral following coronary angiography in Sweden.

DESIGN

Prospective survey and review of medical records.

PATIENTS

Consecutive series of 2767 patients who underwent coronary angiography in Sweden between May 1994 and January 1995 and were considered for coronary revascularisation.

MAIN OUTCOME MEASURES

Percentage of patients referred for coronary artery bypass graft surgery (CABG) and percutaneous transluminal coronary angioplasty (PTCA) for indications that were judged necessary, appropriate, uncertain, and inappropriate by a multispecialty Swedish national expert panel using the RAND/University of California Los Angeles (UCLA) appropriateness method, and the percentage of patients referred for continued medical management who met necessity criteria for revascularisation.

RESULTS

Half the patients were referred for CABG, 25% for PTCA, and 25% for continued medical therapy. CABG was judged appropriate or necessary for 78% of patients, uncertain for 12% and inappropriate for 10%. For PTCA the figures were 32%, 30% and 38%, respectively. Two factors contributed to the high inappropriate rate. Many of these patients did not have "significant" coronary artery disease (although all had at least one stenosis > 50%) or they were treated with less than "optimal" medical therapy. While 96% of patients who met necessity criteria for revascularisation were appropriately referred for revascularisation, 4% were referred for continued medical therapy.

CONCLUSIONS

Using the RAND/UCLA appropriateness method and the definitions agreed to by the expert panel, which may be considered conservative today, it was found that 19% of Swedish patients were referred for coronary revascularisation judged inappropriate. Since some cardiovascular procedures evolve rapidly, the proportion of patients referred for inappropriate indications today remains unknown. Nevertheless, physicians should actively identify those patients who will and will not benefit from coronary revascularisation and ensure that they are appropriately treated.

Evaluation of coronary flow velocity dynamics and flow reserve in patients with Kawasaki disease by means of a Doppler guide wire

OBJECTIVES

To assess the pathophysiologic effects of the coronary sequelae of Kawasaki disease on coronary hemodynamic variables, we regionally evaluated the flow velocity dynamics and flow reserve in coronary vessels with lesions using an intracoronary Doppler flow guide wire.

BACKGROUND

The pathophysiologic effects of the coronary sequelae of Kawasaki disease on coronary hemodynamic variables have not been completely clarified, and we previously reported some discrepancies between coronary angiographic findings and exercise stress tests in Kawasaki disease.

METHODS

Doppler phasic coronary flow velocity was determined using an 0.018-in. (0.046-cm) intracoronary Doppler flow guide wire at rest and during the adenosine triphosphate-induced hyperemic response in 95 patients (75 male, 20 female, mean age 9.8+/-6.2 years) with Kawasaki disease.

RESULTS

In 25 patients with coronary aneurysms in 29 vessels, the average peak velocity and diastolic to systolic velocity ratio were significantly (p < 0.05) decreased in the moderate-sized and large-sized aneurysms. Significantly lower values in coronary flow reserve (CFR) were noted in 3 of 10 vessels with moderate aneurysms and in 4 of 7 vessels with large aneurysms. A significant positive correlation (y = 0.53x + 14.6, r2 = 0.91) was observed between the percent diameter stenosis evaluated by angiography and that calculated from the flow velocity measurement. However, the percent diameter stenosis calculated from the flow velocity measurement was underestimated compared with that determined by angiography in the stenotic lesions of intermediate severity. A reduced CFR was noted in five of seven vessels with intermediate stenosis ranging from 50% to 75%, and also in three vessels with mild stenosis ranging from 30% to 40%. A reduced CFR was also observed in six of the eight angiographically normal vessels associated with the area of reduced perfusion on exercise thallium-201 myocardial scintigraphy.

CONCLUSIONS

Abnormalities in flow dynamics and a reduction in flow reserve were revealed in coronary aneurysms of intermediate to large size and in stenotic lesions, even of mild to intermediate severity, in patients with Kawasaki disease. Abnormalities in the coronary microcirculation, as well as epicardial lesions, contribute to the pathophysiologic responses in Kawasaki disease.

Comparison of centralized versus "site-based" measurement of angiographic stenosis for eligibility in the asymptomatic carotid atherosclerosis study

RATIONALE AND OBJECTIVES

The authors determine the reliability of centralized versus noncentralized (site-based) measurement of angiographic stenosis of patients enrolled into the multicenter, prospective, Asymptomatic Carotid Atherosclerosis Study by angiographic studies.

METHODS

Percent agreements and correlations of 244 masked and prospectively interpreted angiograms were calculated for comparison of centralized and noncentralized readers measuring the percent carotid stenosis from the same angiographic studies. Univariate summary statistics for differences in percent stenoses were calculated for these readings.

RESULTS

Agreement between readings were 88.5% and 91.8% with kappa statistics of 0.77 and 0.73 for > or = 60% and > or = 80% stenosis, respectively, for comparison of 33 centers to the designated central reader. Comparison between the designated central reader and a second central reader derived percent agreements of 85.0% and 86.5% with kappa statistics of 0.69 and 0.41 for > or = 60% and > or = 80% stenoses, respectively, for arteries selected from the original group. Hence, agreement was slightly better between the enrolling centers and the designated central reader than between the two central readers.

CONCLUSIONS

Both centralized and noncentralized (site-based) methods of angiographic measurement of stenosis are equally reliable for large, prospective, masked, multicenter trials when quality control measures are instituted to ensure uniform application of eligibility criteria.

Clinical outcome of deferring angioplasty in patients with normal translesional pressure-flow velocity measurements

OBJECTIVES

The objective of this study was to determine the feasibility, safety and outcome of deferring angioplasty in patients with angiographically intermediate lesions that are found not to limit flow, as determined by direct translesional hemodynamic assessment.

BACKGROUND

The clinical importance of some coronary stenoses of intermediate angiographic severity frequently requires noninvasive stress testing. Direct translesional pressure and flow measurements may assist in clinical decision making in patients with such stenoses.

METHODS

Translesional spectral flow velocity (Doppler guide wire) and pressure data were obtained in 88 patients for 100 lesions (26 single-vessel and 74 multivessel coronary artery lesions) with quantitative angiographic coronary narrowings (mean +/- SD diameter narrowing 54 +/- 7% [range 40% to 74%]). Target lesion angioplasty was prospectively deferred on the basis of predetermined normal values, defined as a proximal/distal velocity ratio < 1.7 or a pressure gradient < 25 mm Hg, or both. Patients were followed up for 9 +/- 5 months (range 6 to 30).

RESULTS

In the deferred angioplasty group, translesional velocity ratios were similar to those of a normal reference group (mean 1.1 +/- 0.32 vs. 1.3 +/- 0.55) and significantly lower than those of a reference cohort of patients who had undergone angioplasty (2.27 +/- 1.2, p < 0.05). The mean translesional pressure gradient in the deferred angioplasty group was also lower than that in the angioplasty group (10 +/- 9 vs. 45 +/- 22 mm Hg, p < 0.001). At follow-up in the deferred angioplasty group, four, six, zero and two patients, respectively, had had subsequent angioplasty, coronary artery bypass graft surgery or myocardial infarction or had died. In one patient, death was related to angioplasty of a nontarget artery lesion, and one patient with multivessel disease had a cardiac arrest due to ventricular fibrillation 12 months after lesion assessment. Among the 10 patients requiring later angioplasty or coronary artery bypass grafting, only six procedures were performed on target arteries. No patient had a complication of translesional flow or pressure measurements.

CONCLUSIONS

These data demonstrate the safety, feasibility and clinical outcome of deferring angioplasty of coronary artery narrowings associated with normal translesional coronary hemodynamic variables. Given the practice of performing angioplasty without ischemic testing or when testing is inconclusive, translesional hemodynamic data obtained at diagnostic catheterization can identify patients in whom it is safe to postpone angioplasty.