Testing the test: remediation is needed

Appeals in gynecologic cytology proficiency testing: review and analysis of data from the 2006 College of American Pathologists gynecologic cytology proficiency testing program

CONTEXT

In 2006, 9643 participants took the initial College of American Pathologists (CAP) Proficiency Test (PT). Failing participants may appeal results on specific test slides. Appeals are granted if 3 referee pathologists do not unanimously agree on the initial reference diagnosis in a masked review process.

OBJECTIVES

To investigate causes of PT failures, subsequent appeals, and appeal successes in 2006.

DESIGN

Appeals were examined, including patient demographic information, Centers for Medicare and Medicaid Services category (A, B, C, or D), exact reference diagnosis, examinees per appeal, examinee's Centers for Medicare and Medicaid Services category, referee's Centers for Medicare and Medicaid Services category, slide preparation type, and slide field validation rate.

RESULTS

There was a 94% passing rate for 2006. One hundred fifty-five examinees (1.6%) appealed 86 slides of all preparation types. Forty-five appeals (29%) were granted on 21 slides; 110 appeals (72%) were denied on 65 slides. Reference category D and B slides were most often appealed. The highest percentage of granted appeals occurred in category D (35% slides; 42% of participants) and the lowest occurred in category B (9% slides; 8% of participants). The field validation rate of all appealed slides was greater than 90%.

CONCLUSIONS

Despite rigorous field validation of slides, 6% of participants failed. Thirty percent of failing participants appealed; most appeals involved misinterpretation of category D as category B. Referees were never unanimous in their agreement with the participant. The participants and referees struggled with the reliability and reproducibility of finding rare cells, "overdiagnosis" of benign changes, and assigning the morphologically dynamic biologic changes of squamous intraepithelial lesions to static categories.