Factors related to inter-observer reproducibility of conventional Pap smear cytology: a multilevel analysis of smear and laboratory characteristics

OBJECTIVE

To identify factors that influence the inter-observer reproducibility of the routine, conventional Pap smear cytology (Pap smear test) in a network of certificated laboratories in a middle-income Latin American country.

METHODS

Twenty-six laboratories provided each an average of 26 negative for malignancy (NILM) and high-grade squamous intraepithelial lesion (HSIL) Pap smears. An external panel reviewed the slides. The kappa index and multilevel logistic regression were used to estimate the reproducibility and odds ratios (OR) of a false result with 95% confidence intervals (95% CI), respectively. Results are presented for laboratories that collect (collector laboratories) and do not collect (non-collector laboratories) samples.

RESULTS

The agreements ranged widely (median kappa 0.51, range 0.16-0.70). The overall false-positive (FP) and false-negative (FN) rates were 31% (95% CI 27-35) and 11% (95% CI 7-17). Among collector laboratories (N = 14), a bigger sample collection volume decreased the probability of a FP (OR-adjusted 0.05, 95% CI 0.02-0.1) whereas the number of quality defects (OR-adjusted 1.67, 95% CI 1.25-2.24), high workload (OR-adjusted 5.52, 95% CI 3.85-7.92) and collection by cytotechnologists (OR-adjusted 1.28, 95% CI 1.15-1.42) or health professionals (OR-adjusted 2.26, 95% CI 2.04-2.49) instead of nursing assistants increased it. Among non-collector laboratories (N = 9), the FP rate increased with the number of quality defects (OR-adjusted 1.86, 95% CI 1.06-3.26) but decreased if the samples were collected by health professionals instead of nursing assistants (OR-adjusted 0.37, 95%CI 0.17-0.80). No significant associations were observed for FN.

CONCLUSIONS

Staff in charge of cervical sampling significantly determined the reproducibility of the Pap smear test, but this depended on whether the laboratory collects samples or read samples collected elsewhere.

Mock gynecologic cytology proficiency testing as a milestone assessment tool for anatomic pathology residents

INTRODUCTION

One of the major aims of the Next Accreditation System is to move toward an outcomes-based evaluation system where each accredited medical residency program must demonstrate that its residents are competent in performing the essential tasks necessary for clinical practice. Because all pathologists who sign-out or screen Papanicolaou (Pap) tests are required to pass an annual 10-slide gynecologic cytology proficiency test (PT), we developed mock PT modules as a tool for assessing competency.

MATERIALS AND METHODS

In 2007, we introduced mock proficiency testing with 3 distinct modules, each consisting of 3 10-slide test sets (10 ThinPrep, 10 SurePath, and 10 conventional Pap slides). Each module was administered at 3 different time points. We evaluated the following parameters: (1) performance differences between Pap preparations; (2) performance over time; (3) performance before and after initiation of one-on-one teaching sessions with cytotechnologists in 2009; and (4) quality of test slides.

RESULTS

Residents showed improvement over time, and overall scores did not differ significantly among ThinPrep, SurePath, and conventional slide sets. The average score for the first test set was significantly higher for residents who received formal training by a cytotechnologist than for those who did not. Overall, 16 of 90 slides were misclassified by 40% or more of residents, half of which exhibited glandular abnormalities.

CONCLUSIONS

The objective assessment provided by mock PT is a useful tool for both faculty and residents.

False-positive Papanicolaou (PAP) test rates in the College of American Pathologists PAP education and PAP proficiency test programs: evaluation of false-positive responses of high-grade squamous intraepithelial lesion or cancer to a negative reference diagnosis

CONTEXT

In cytology proficiency testing (PT), participants fail for incorrectly interpreting a high-grade squamous intraepithelial lesion or cancer (HSIL+) Papanicolaou test result as negative. This penalty may lead to a false-positive interpretation of negative slides as HSIL+ to avoid failure.

OBJECTIVE

To investigate factors related to false-positive responses in a PT versus an educational environment.

DESIGN

We analyzed 420,079 responses from 9414 validated negative reference slides in the College of American Pathologists Interlaboratory Comparison Program in Gynecologic Cytopathology (PAP Education) and compared them with responses from the Gynecologic Cytology Proficiency Testing Program for the percentage of false-positive (HSIL+) interpretations in each of 7 negative subcategories. We evaluated the influence of preparation type (ThinPrep, SurePath, and conventional Papanicolaou test), participant type (pathologist or cytotechnologist), and program time interval (preproficiency test or PT) on a false-positive response.

RESULTS

Reference diagnosis and participant type, but not preparation type, were statistically correlated to false-positive responses. The interaction between program time interval and participant type was also significant. Pathologists had higher rates of false-positive results on preproficiency test (1.2% [800 of 68,690]) than they did on PT (0.8% [993 of 129,857]). Cytotechnologists had no differences between program time intervals (preproficiency, 0.9% [515 of 63,281] versus PT, 1.0 [1231 of 121,621]; P = .91). Negative subcategories frequently mistaken for HSIL+ were reparative changes (4.7% [427 of 9069]), atrophic vaginitis (1.8% [18 of 987]), and negative for intraepithelial lesion or malignancy (1.2% [2143 of 178,651]), but during PT, false-positive rates were significantly increased only for the negative for intraepithelial lesion or malignancy and herpes simplex virus (P < .001).

CONCLUSIONS

Pathologists had lower false-positive rates in the Gynecologic Cytology Proficiency Testing Program than they did in PAP Education, but participants were more likely to report a false-positive response (HSIL+) for negative for intraepithelial lesion or malignancy and herpes simplex virus in the Gynecologic Cytology Proficiency Test Program.

Reducing false-negative and false-positive diagnoses in anatomic pathology consultation material

CONTEXT

Previous studies have shown that there are disagreements in interlaboratory consultation, including false-negative and false-positive diagnoses. To date, methods to reduce false-negative and false-positive diagnoses have been poorly documented.

OBJECTIVE

To identify features associated with false-negative and false-positive diagnoses in anatomic pathology.

DESIGN

We reviewed the results of interlaboratory consultation in our institution during a 9-year period. For false-negative and false-positive diagnoses, methods that might have prevented the error were identified.

RESULTS

Disagreements were identified in 810 of 8082 consults (10%). Fifty-four false-negative cases (0.7% of all consults) and 27 false-positive cases (0.3%) were identified. False-negative cases were more common in breast (20 of 1131; 1.8%), genitourinary (16 of 970; 1.7%), hematologic (3 of 242; 1.3%), and cytology (3 of 404; 0.8%) than in all other sites combined (P < .001); no significant difference in sites were identified for false-positive cases. Overall, there was no difference in the percentage of cases that were reviewed by more than one pathologist in either false-negative cases (109 of 810; 13.5%) or false-positive cases (135 of 810; 16.7%), compared with all other consults (858 of 7272; 11.8%) (P = .74 and .59, respectively). However, on review, 12 of all 27 false-positive cases (44%) might have been prevented by the use of immunohistochemistry alone, and 36 of all 54 false-negative cases (67%) might have been prevented by the use of a second review; special stains, including immunohistochemistry; additional levels; changes in processing; and hedges.

CONCLUSION

Approximately one-half of false-negative and false-positive cases (48 of 81; 59%) might be preventable by the use of a combination of pathologic methods.